[==========] Running 56 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 30 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:21:01 dist-test-slave-h3ft krb5kdc[16737](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:21:01 dist-test-slave-h3ft krb5kdc[16737](info): set up 2 sockets
Aug 06 09:21:01 dist-test-slave-h3ft krb5kdc[16737](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260806 09:21:02.984438 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.341s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:21:03 dist-test-slave-h3ft krb5kdc[16737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:21:03Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:03Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:21:03.257608 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:36063
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:36063
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:03.699409 16758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:03.699851 16758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:03.700049 16758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:03.709393 16758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:03.709623 16758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:03.709736 16758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:03.709952 16758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:03.710067 16758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:21:03.722895 16758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:36063
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:36063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:03.725024 16758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:03.726864 16758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:03.743376 16765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:03.743512 16764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:03.743429 16767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:03.745457 16758 server_base.cc:1034] running on GCE node
I20260806 09:21:03.746726 16758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:03.748988 16758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:03.752424 16758 hybrid_clock.cc:648] HybridClock initialized: now 1786008063752371 us; error 147 us; skew 500 ppm
I20260806 09:21:03.753043 16758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:03 dist-test-slave-h3ft krb5kdc[16737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008063, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:03.757603 16758 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:21:03.760020 16758 webserver.cc:460] Webserver started at http://127.16.86.190:41953/ using document root <none> and password file <none>
I20260806 09:21:03.761054 16758 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:03.761236 16758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:03.761654 16758 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:03.766000 16758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "846ad349dfe5498da572684c2256f79d"
format_stamp: "Formatted at 2026-08-06 09:21:03 on dist-test-slave-h3ft"
server_key: "78ff60298cbeeb4b0830070c99b96670"
server_key_iv: "95a5aa734472e7ea814f7433e951bcee"
server_key_version: "encryptionkey@0"
I20260806 09:21:03.767174 16758 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "846ad349dfe5498da572684c2256f79d"
format_stamp: "Formatted at 2026-08-06 09:21:03 on dist-test-slave-h3ft"
server_key: "78ff60298cbeeb4b0830070c99b96670"
server_key_iv: "95a5aa734472e7ea814f7433e951bcee"
server_key_version: "encryptionkey@0"
I20260806 09:21:03.775277 16758 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260806 09:21:03.781091 16779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:03.783196 16758 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:21:03.783488 16758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "846ad349dfe5498da572684c2256f79d"
format_stamp: "Formatted at 2026-08-06 09:21:03 on dist-test-slave-h3ft"
server_key: "78ff60298cbeeb4b0830070c99b96670"
server_key_iv: "95a5aa734472e7ea814f7433e951bcee"
server_key_version: "encryptionkey@0"
I20260806 09:21:03.783769 16758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:03.812681 16758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:03.819094 16758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:03.819507 16758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:03.847177 16758 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:36063
I20260806 09:21:03.847172 16848 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:36063 every 8 connection(s)
I20260806 09:21:03.849740 16758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:03.856014 16849 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:03.858523 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 16758
I20260806 09:21:03.859288 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:03.863106 16730 external_mini_cluster.cc:1428] Setting key 52d54a03a694c161221a2d26b3934c5a
I20260806 09:21:03.873437 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d: Bootstrap starting.
I20260806 09:21:03.877347 16849 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:03.878513 16849 log.cc:824] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:03.882714 16849 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d: No bootstrap required, opened a new log
I20260806 09:21:03.893790 16849 raft_consensus.cc:348] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846ad349dfe5498da572684c2256f79d" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36063 } }
I20260806 09:21:03.894389 16849 raft_consensus.cc:374] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:03.894555 16849 raft_consensus.cc:729] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 846ad349dfe5498da572684c2256f79d, State: Initialized, Role: FOLLOWER
I20260806 09:21:03.895383 16849 consensus_queue.cc:260] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846ad349dfe5498da572684c2256f79d" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36063 } }
I20260806 09:21:03.895674 16849 raft_consensus.cc:388] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:03.895853 16849 raft_consensus.cc:482] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:03.896057 16849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:03.899308 16849 raft_consensus.cc:504] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846ad349dfe5498da572684c2256f79d" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36063 } }
I20260806 09:21:03.900054 16849 leader_election.cc:302] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 846ad349dfe5498da572684c2256f79d; no voters:
I20260806 09:21:03.901149 16849 leader_election.cc:288] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:03.901448 16855 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:03.903548 16855 raft_consensus.cc:686] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [term 1 LEADER]: Becoming Leader. State: Replica: 846ad349dfe5498da572684c2256f79d, State: Running, Role: LEADER
I20260806 09:21:03.904709 16855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846ad349dfe5498da572684c2256f79d" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36063 } }
I20260806 09:21:03.905547 16849 sys_catalog.cc:564] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 09:21:03 dist-test-slave-h3ft krb5kdc[16737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:03.909081 16857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 846ad349dfe5498da572684c2256f79d. Latest consensus state: current_term: 1 leader_uuid: "846ad349dfe5498da572684c2256f79d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846ad349dfe5498da572684c2256f79d" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36063 } } }
I20260806 09:21:03.909610 16857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:03.910271 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "846ad349dfe5498da572684c2256f79d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "846ad349dfe5498da572684c2256f79d" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36063 } } }
I20260806 09:21:03.910660 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:03.921527 16861 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:03.929975 16852 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:03.870303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33292 (local address 127.16.86.190:36063)
0806 09:21:03.871730 (+ 1427us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:03.871751 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:03.873837 (+ 2086us) server_negotiation.cc:407] Connection header received
0806 09:21:03.885747 (+ 11910us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:03.885791 (+ 44us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:03.886400 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:03.886948 (+ 548us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:03.891805 (+ 4857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:03.894137 (+ 2332us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:03.897127 (+ 2990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:03.898365 (+ 1238us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:03.898649 (+ 284us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:03.910812 (+ 12163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:03.910831 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:03.910898 (+ 67us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:03.910985 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:03.918231 (+ 7246us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:03.920145 (+ 1914us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:03.920160 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:03.920173 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:03.920508 (+ 335us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:03.921584 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:03.921592 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:03.921598 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:03.922925 (+ 1327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:03.926818 (+ 3893us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:03.927314 (+ 496us) server_negotiation.cc:299] Negotiation successful
0806 09:21:03.928026 (+ 712us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1088,"thread_start_us":707,"threads_started":1}
I20260806 09:21:03.932461 16861 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:03.946317 16861 catalog_manager.cc:1347] Generated new cluster ID: ff09b389966b4b99b41b141bce5c3160
I20260806 09:21:03.946486 16861 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:03.966722 16861 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:03.968699 16861 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:03.978791 16861 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 846ad349dfe5498da572684c2256f79d: Generated new TSK 0
I20260806 09:21:03.980443 16861 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:21:04.053177 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36063
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:04.523502 16883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:04.523906 16883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:04.524027 16883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:04.532208 16883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:04.532366 16883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:04.532502 16883 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:21:04.544584 16883 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36063
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:04.546384 16883 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:04.548306 16883 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:04.567283 16889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:04.567256 16890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:04.567171 16892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:04.574157 16883 server_base.cc:1034] running on GCE node
I20260806 09:21:04.575474 16883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:04.577028 16883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:04.578369 16883 hybrid_clock.cc:648] HybridClock initialized: now 1786008064578267 us; error 94 us; skew 500 ppm
I20260806 09:21:04.578817 16883 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:04 dist-test-slave-h3ft krb5kdc[16737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008064, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:04.583909 16883 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:21:04.586272 16883 webserver.cc:460] Webserver started at http://127.16.86.129:34279/ using document root <none> and password file <none>
I20260806 09:21:04.587440 16883 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:04.587638 16883 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:04.588079 16883 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:04.592221 16883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "2925acac293a45f6ac57ff16b83a6698"
format_stamp: "Formatted at 2026-08-06 09:21:04 on dist-test-slave-h3ft"
server_key: "d6533134efe20e823bb4ca1e6b5f4ad5"
server_key_iv: "1d56f04579e29cd6b034251527eacc00"
server_key_version: "encryptionkey@0"
I20260806 09:21:04.593350 16883 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "2925acac293a45f6ac57ff16b83a6698"
format_stamp: "Formatted at 2026-08-06 09:21:04 on dist-test-slave-h3ft"
server_key: "d6533134efe20e823bb4ca1e6b5f4ad5"
server_key_iv: "1d56f04579e29cd6b034251527eacc00"
server_key_version: "encryptionkey@0"
I20260806 09:21:04.601723 16883 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260806 09:21:04.607152 16902 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:04.609336 16883 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:21:04.609666 16883 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "2925acac293a45f6ac57ff16b83a6698"
format_stamp: "Formatted at 2026-08-06 09:21:04 on dist-test-slave-h3ft"
server_key: "d6533134efe20e823bb4ca1e6b5f4ad5"
server_key_iv: "1d56f04579e29cd6b034251527eacc00"
server_key_version: "encryptionkey@0"
I20260806 09:21:04.609987 16883 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:04.630784 16883 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:04.636469 16883 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:04.636848 16883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:04.638260 16883 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:04.640072 16883 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:04.640193 16883 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:04.640413 16883 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:04.640496 16883 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:04.689502 16883 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:35009
I20260806 09:21:04.689543 17059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:35009 every 8 connection(s)
I20260806 09:21:04.691392 16883 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:21:04.696012 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 16883
I20260806 09:21:04.696352 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:21:04.697245 16730 external_mini_cluster.cc:1428] Setting key fc791b1ec5c824a8119ee034417560ff
Aug 06 09:21:04 dist-test-slave-h3ft krb5kdc[16737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008064, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
I20260806 09:21:04.725598 17063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:04.695670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:36063 (local address 127.16.86.129:59927)
0806 09:21:04.696989 (+ 1319us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:04.697047 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:04.699664 (+ 2617us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:04.700605 (+ 941us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:04.700616 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:04.701332 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:04.702117 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:04.702139 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:04.703486 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:04.703490 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:04.705158 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:04.705174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:04.706131 (+ 957us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:04.706140 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:04.706573 (+ 433us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:04.707705 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:04.707734 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:04.711356 (+ 3622us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:04.718494 (+ 7138us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:04.718510 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:04.718554 (+ 44us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:04.719655 (+ 1101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:04.720283 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:04.720290 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:04.720294 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:04.720701 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:04.721714 (+ 1013us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:04.721729 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:04.722539 (+ 810us) client_negotiation.cc:769] Sending connection context
0806 09:21:04.722983 (+ 444us) client_negotiation.cc:240] Negotiation successful
0806 09:21:04.723711 (+ 728us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":938,"thread_start_us":625,"threads_started":1}
I20260806 09:21:04.727345 17062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:04.695121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:59927 (local address 127.16.86.190:36063)
0806 09:21:04.695725 (+ 604us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:04.695730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:04.697231 (+ 1501us) server_negotiation.cc:407] Connection header received
0806 09:21:04.700009 (+ 2778us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:04.700014 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:04.700133 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:04.700272 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:04.702336 (+ 2064us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:04.703280 (+ 944us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:04.705348 (+ 2068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:04.705816 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:04.705998 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:04.711773 (+ 5775us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:04.711780 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:04.711783 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:04.711861 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:04.718105 (+ 6244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:04.719911 (+ 1806us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:04.719916 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:04.719920 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:04.720119 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:04.720898 (+ 779us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:04.720905 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:04.720911 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:04.721525 (+ 614us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:04.726296 (+ 4771us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:04.726536 (+ 240us) server_negotiation.cc:299] Negotiation successful
0806 09:21:04.726831 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":340,"threads_started":1}
I20260806 09:21:04.729215 17060 heartbeater.cc:347] Connected to a master server at 127.16.86.190:36063
I20260806 09:21:04.729676 17060 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:04.730670 17060 heartbeater.cc:511] Master 127.16.86.190:36063 requested a full tablet report, sending...
I20260806 09:21:04.733999 16800 ts_manager.cc:194] Registered new tserver with Master: 2925acac293a45f6ac57ff16b83a6698 (127.16.86.129:35009)
I20260806 09:21:04.736627 16800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:59927
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:21:04.763451 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36063
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:05.246452 17066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:05.246897 17066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:05.247098 17066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:05.255820 17066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:05.256000 17066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:05.256143 17066 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:21:05.267757 17066 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36063
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:05.269773 17066 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:05.271597 17066 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:05.292148 17076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:05.292237 17074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:05.292116 17073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:05.294435 17066 server_base.cc:1034] running on GCE node
I20260806 09:21:05.295511 17066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:05.297058 17066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:05.298422 17066 hybrid_clock.cc:648] HybridClock initialized: now 1786008065298356 us; error 58 us; skew 500 ppm
I20260806 09:21:05.298938 17066 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:05 dist-test-slave-h3ft krb5kdc[16737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:05.303578 17066 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:21:05.305949 17066 webserver.cc:460] Webserver started at http://127.16.86.130:44683/ using document root <none> and password file <none>
I20260806 09:21:05.307053 17066 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:05.307247 17066 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:05.307669 17066 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:05.311776 17066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "7c3eaa87f38b4efa9d8f8d292dca7040"
format_stamp: "Formatted at 2026-08-06 09:21:05 on dist-test-slave-h3ft"
server_key: "deeb74c9f003d45c456d7fe04d054264"
server_key_iv: "39c3b2ce89003cd3699dfbd564a07bc4"
server_key_version: "encryptionkey@0"
I20260806 09:21:05.312927 17066 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "7c3eaa87f38b4efa9d8f8d292dca7040"
format_stamp: "Formatted at 2026-08-06 09:21:05 on dist-test-slave-h3ft"
server_key: "deeb74c9f003d45c456d7fe04d054264"
server_key_iv: "39c3b2ce89003cd3699dfbd564a07bc4"
server_key_version: "encryptionkey@0"
I20260806 09:21:05.320657 17066 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.002s
I20260806 09:21:05.326057 17088 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:05.328444 17066 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.001s
I20260806 09:21:05.328711 17066 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "7c3eaa87f38b4efa9d8f8d292dca7040"
format_stamp: "Formatted at 2026-08-06 09:21:05 on dist-test-slave-h3ft"
server_key: "deeb74c9f003d45c456d7fe04d054264"
server_key_iv: "39c3b2ce89003cd3699dfbd564a07bc4"
server_key_version: "encryptionkey@0"
I20260806 09:21:05.328997 17066 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:05.356715 17066 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:05.362675 17066 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:05.363147 17066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:05.364629 17066 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:05.366627 17066 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:05.366756 17066 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:05.366925 17066 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:05.367007 17066 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:05.414541 17066 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:35543
I20260806 09:21:05.414669 17235 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:35543 every 8 connection(s)
I20260806 09:21:05.416558 17066 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:21:05.426301 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 17066
I20260806 09:21:05.426508 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:21:05.427057 16730 external_mini_cluster.cc:1428] Setting key f4c15ee3da29fe766f4755ca672f684e
Aug 06 09:21:05 dist-test-slave-h3ft krb5kdc[16737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:05.445448 17240 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:05.420195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:37657 (local address 127.16.86.190:36063)
0806 09:21:05.420877 (+ 682us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:05.420885 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:05.422094 (+ 1209us) server_negotiation.cc:407] Connection header received
0806 09:21:05.424763 (+ 2669us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:05.424771 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:05.424881 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:05.425030 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:05.427634 (+ 2604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:05.428588 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:05.431458 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:05.432057 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:05.432221 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:05.438131 (+ 5910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:05.438138 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:05.438142 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:05.438187 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:05.441442 (+ 3255us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:05.442464 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:05.442469 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:05.442471 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:05.442605 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:05.443182 (+ 577us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:05.443186 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:05.443188 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:05.443510 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:05.443693 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:05.444722 (+ 1029us) server_negotiation.cc:299] Negotiation successful
0806 09:21:05.445015 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":335,"threads_started":1}
I20260806 09:21:05.446796 17242 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:05.420611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:36063 (local address 127.16.86.130:37657)
0806 09:21:05.421778 (+ 1167us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:05.421857 (+ 79us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:05.424245 (+ 2388us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:05.425413 (+ 1168us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:05.425425 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:05.426238 (+ 813us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:05.427328 (+ 1090us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:05.427355 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:05.428875 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:05.428882 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:05.431208 (+ 2326us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:05.431236 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:05.432380 (+ 1144us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:05.432393 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:05.432842 (+ 449us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:05.433958 (+ 1116us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:05.434137 (+ 179us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:05.437763 (+ 3626us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:05.441660 (+ 3897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:05.441674 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:05.441700 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:05.442295 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:05.442732 (+ 437us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:05.442737 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:05.442741 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:05.442981 (+ 240us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:05.443671 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:05.443682 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:05.444237 (+ 555us) client_negotiation.cc:769] Sending connection context
0806 09:21:05.444602 (+ 365us) client_negotiation.cc:240] Negotiation successful
0806 09:21:05.445313 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":452,"threads_started":1}
I20260806 09:21:05.449126 17236 heartbeater.cc:347] Connected to a master server at 127.16.86.190:36063
I20260806 09:21:05.449676 17236 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
I20260806 09:21:05.450801 17236 heartbeater.cc:511] Master 127.16.86.190:36063 requested a full tablet report, sending...
I20260806 09:21:05.453136 16800 ts_manager.cc:194] Registered new tserver with Master: 7c3eaa87f38b4efa9d8f8d292dca7040 (127.16.86.130:35543)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
I20260806 09:21:05.454408 16800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:37657
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:21:05.491613 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36063
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:43765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:21:05.742161 17060 heartbeater.cc:503] Master 127.16.86.190:36063 was elected leader, sending a full tablet report...
W20260806 09:21:05.950734 17247 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:05.951232 17247 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:05.951418 17247 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:05.959761 17247 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:05.959980 17247 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:05.960196 17247 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:21:05.972505 17247 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:43765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36063
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:05.974406 17247 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:05.976256 17247 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:05.993221 17256 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:05.993523 17254 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:05.993574 17253 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:05.994995 17247 server_base.cc:1034] running on GCE node
I20260806 09:21:05.995942 17247 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:05.997295 17247 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:05.998575 17247 hybrid_clock.cc:648] HybridClock initialized: now 1786008065998491 us; error 79 us; skew 500 ppm
I20260806 09:21:05.999089 17247 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:05 dist-test-slave-h3ft krb5kdc[16737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:06.003279 17247 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:21:06.005376 17247 webserver.cc:460] Webserver started at http://127.16.86.131:39665/ using document root <none> and password file <none>
I20260806 09:21:06.006353 17247 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:06.006531 17247 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:06.006989 17247 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:06.010753 17247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "6ed4a2ee3f32431aa4a4f561a063567d"
format_stamp: "Formatted at 2026-08-06 09:21:06 on dist-test-slave-h3ft"
server_key: "3e61f08f467b32eff1b1415491e47941"
server_key_iv: "5736cb40f249b204ebc0f59844ff8dd2"
server_key_version: "encryptionkey@0"
I20260806 09:21:06.011893 17247 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "6ed4a2ee3f32431aa4a4f561a063567d"
format_stamp: "Formatted at 2026-08-06 09:21:06 on dist-test-slave-h3ft"
server_key: "3e61f08f467b32eff1b1415491e47941"
server_key_iv: "5736cb40f249b204ebc0f59844ff8dd2"
server_key_version: "encryptionkey@0"
I20260806 09:21:06.018759 17247 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 09:21:06.023537 17271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:06.025462 17247 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:21:06.025713 17247 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "6ed4a2ee3f32431aa4a4f561a063567d"
format_stamp: "Formatted at 2026-08-06 09:21:06 on dist-test-slave-h3ft"
server_key: "3e61f08f467b32eff1b1415491e47941"
server_key_iv: "5736cb40f249b204ebc0f59844ff8dd2"
server_key_version: "encryptionkey@0"
I20260806 09:21:06.026002 17247 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:06.055415 17247 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:06.061097 17247 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:06.061506 17247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:06.062827 17247 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:06.064615 17247 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:06.064735 17247 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:06.064914 17247 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:06.064996 17247 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:06.111497 17247 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:37317
I20260806 09:21:06.111582 17419 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:37317 every 8 connection(s)
I20260806 09:21:06.113336 17247 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:21:06.122802 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 17247
I20260806 09:21:06.123092 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:21:06.123693 16730 external_mini_cluster.cc:1428] Setting key 144bdaa56c5118c5db9b6b7ebbce536b
Aug 06 09:21:06 dist-test-slave-h3ft krb5kdc[16737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008065, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:06.141654 17423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:06.116713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:47693 (local address 127.16.86.190:36063)
0806 09:21:06.117356 (+ 643us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:06.117361 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:06.118427 (+ 1066us) server_negotiation.cc:407] Connection header received
0806 09:21:06.120861 (+ 2434us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:06.120868 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:06.120989 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:06.121131 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:06.123225 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.124179 (+ 954us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.126329 (+ 2150us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.126776 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.126945 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:06.132813 (+ 5868us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:06.132820 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:06.132823 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:06.132869 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:06.136975 (+ 4106us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:06.138130 (+ 1155us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:06.138135 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:06.138139 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:06.138309 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:06.138879 (+ 570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:06.138883 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:06.138886 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:06.139306 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:06.139548 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:06.140820 (+ 1272us) server_negotiation.cc:299] Negotiation successful
0806 09:21:06.141175 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":512,"thread_start_us":397,"threads_started":1}
I20260806 09:21:06.143038 17424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:06.117215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:36063 (local address 127.16.86.131:47693)
0806 09:21:06.118218 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:06.118269 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:06.120469 (+ 2200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:06.121451 (+ 982us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:06.121466 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:06.122160 (+ 694us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:06.122955 (+ 795us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:06.122986 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.124352 (+ 1366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.124359 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:06.126165 (+ 1806us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:06.126186 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.127225 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.127239 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:06.127833 (+ 594us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:06.128945 (+ 1112us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:06.128984 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:06.132477 (+ 3493us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:06.137245 (+ 4768us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:06.137257 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:06.137279 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:06.137947 (+ 668us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:06.138451 (+ 504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:06.138457 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:06.138462 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:06.138716 (+ 254us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:06.139467 (+ 751us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:06.139480 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:06.140014 (+ 534us) client_negotiation.cc:769] Sending connection context
0806 09:21:06.140449 (+ 435us) client_negotiation.cc:240] Negotiation successful
0806 09:21:06.141165 (+ 716us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":464,"threads_started":1}
I20260806 09:21:06.145083 17420 heartbeater.cc:347] Connected to a master server at 127.16.86.190:36063
I20260806 09:21:06.145466 17420 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:06.146520 17420 heartbeater.cc:511] Master 127.16.86.190:36063 requested a full tablet report, sending...
I20260806 09:21:06.148669 16800 ts_manager.cc:194] Registered new tserver with Master: 6ed4a2ee3f32431aa4a4f561a063567d (127.16.86.131:37317)
I20260806 09:21:06.149874 16800 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:47693
I20260806 09:21:06.157284 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:21:06 dist-test-slave-h3ft krb5kdc[16737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:21:06 dist-test-slave-h3ft krb5kdc[16737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008066, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:21:06.224202 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:06.203066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55720 (local address 127.16.86.129:35009)
0806 09:21:06.204148 (+ 1082us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:06.204158 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:06.204198 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:21:06.204292 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:06.204301 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:06.204598 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:06.204779 (+ 181us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:06.206851 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.208289 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.210033 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.210779 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.210933 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:06.216474 (+ 5541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:06.216486 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:06.216502 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:06.216553 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:06.220256 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:06.221297 (+ 1041us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:06.221305 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:06.221307 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:06.221466 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:06.222079 (+ 613us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:06.222094 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:06.222097 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:06.222685 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:06.222983 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:06.223579 (+ 596us) server_negotiation.cc:299] Negotiation successful
0806 09:21:06.223864 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":876,"thread_start_us":630,"threads_started":1}
W20260806 09:21:06.226446 16947 server_base.cc:1116] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:55720
Aug 06 09:21:06 dist-test-slave-h3ft krb5kdc[16737](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:21:06 dist-test-slave-h3ft krb5kdc[16737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:21:06.292380 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:06.274567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55728 (local address 127.16.86.129:35009)
0806 09:21:06.274910 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:06.274923 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:06.275318 (+ 395us) server_negotiation.cc:407] Connection header received
0806 09:21:06.275559 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:06.275570 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:06.275652 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:06.275768 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:06.277303 (+ 1535us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.278044 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.279386 (+ 1342us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.279991 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.280147 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:06.285811 (+ 5664us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:06.285844 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:06.285853 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:06.285920 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:06.289522 (+ 3602us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:06.290288 (+ 766us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:06.290298 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:06.290302 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:06.290448 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:06.290833 (+ 385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:06.290837 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:06.290839 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:06.291176 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:06.291459 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:06.291767 (+ 308us) server_negotiation.cc:299] Negotiation successful
0806 09:21:06.292018 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260806 09:21:06.294804 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 16883
I20260806 09:21:06.311412 17430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:06.296356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55734 (local address 127.16.86.129:35009)
0806 09:21:06.296632 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:06.296641 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:06.297086 (+ 445us) server_negotiation.cc:407] Connection header received
0806 09:21:06.297310 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:06.297317 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:06.297388 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:06.297492 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:06.298810 (+ 1318us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.299601 (+ 791us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.301318 (+ 1717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.301939 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.302136 (+ 197us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:06.305133 (+ 2997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:06.305139 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:06.305142 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:06.305176 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:06.308486 (+ 3310us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:06.309222 (+ 736us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:06.309230 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:06.309234 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:06.309390 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:06.309902 (+ 512us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:06.309909 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:06.309915 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:06.310212 (+ 297us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:06.310472 (+ 260us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:06.310765 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 09:21:06.310986 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 09:21:06.312203 17055 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:06.410084 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 16883
I20260806 09:21:06.445309 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 17066
Aug 06 09:21:06 dist-test-slave-h3ft krb5kdc[16737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:21:06.457849 17236 heartbeater.cc:503] Master 127.16.86.190:36063 was elected leader, sending a full tablet report...
I20260806 09:21:06.466221 17439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:06.446597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46246 (local address 127.16.86.130:35543)
0806 09:21:06.447489 (+ 892us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:06.447499 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:06.447535 (+ 36us) server_negotiation.cc:407] Connection header received
0806 09:21:06.447616 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:06.447625 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:06.447938 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:06.448157 (+ 219us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:06.449729 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.450947 (+ 1218us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.452712 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.453451 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.453618 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:06.459238 (+ 5620us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:06.459254 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:06.459275 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:06.459318 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:06.462512 (+ 3194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:06.463340 (+ 828us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:06.463347 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:06.463349 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:06.463488 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:06.464124 (+ 636us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:06.464131 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:06.464134 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:06.464605 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:06.464977 (+ 372us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:06.465346 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 09:21:06.465678 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":662,"thread_start_us":440,"threads_started":1}
I20260806 09:21:06.468034 17231 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:06.559057 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 17066
I20260806 09:21:06.584353 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 17247
Aug 06 09:21:06 dist-test-slave-h3ft krb5kdc[16737](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008066, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:21:06.604465 17441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:06.585599 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53938 (local address 127.16.86.131:37317)
0806 09:21:06.586340 (+ 741us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:06.586350 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:06.586392 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:21:06.586475 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:06.586483 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:06.586774 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:06.587043 (+ 269us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:06.588542 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.589789 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.591282 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:06.591913 (+ 631us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:06.592157 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:06.597237 (+ 5080us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:06.597251 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:06.597272 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:06.597315 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:06.600949 (+ 3634us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:06.601814 (+ 865us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:06.601822 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:06.601825 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:06.601986 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:06.602415 (+ 429us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:06.602419 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:06.602421 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:06.602868 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:06.603296 (+ 428us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:06.603650 (+ 354us) server_negotiation.cc:299] Negotiation successful
0806 09:21:06.604006 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":307,"threads_started":1}
I20260806 09:21:06.606423 17414 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:06.697584 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 17247
I20260806 09:21:06.724294 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 16758
I20260806 09:21:06.725898 16842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:06.835083 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 16758
2026-08-06T09:21:06Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (5264 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:21:06 dist-test-slave-h3ft krb5kdc[17447](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:21:06 dist-test-slave-h3ft krb5kdc[17447](info): set up 2 sockets
Aug 06 09:21:06 dist-test-slave-h3ft krb5kdc[17447](info): commencing operation
krb5kdc: starting...
W20260806 09:21:08.157260 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.261s user 0.005s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:21:08 dist-test-slave-h3ft krb5kdc[17447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:21:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:21:08.411696 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:35805
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:35805
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:08.824174 17464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:08.824512 17464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:08.824623 17464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:08.832930 17464 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:08.833062 17464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:08.833117 17464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:08.833187 17464 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:08.833243 17464 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:21:08.846199 17464 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:35805
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:35805
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:08.848122 17464 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:08.850131 17464 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:08.865299 17479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:08.865509 17476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:08.865869 17472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:08.866745 17464 server_base.cc:1034] running on GCE node
I20260806 09:21:08.867741 17464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:08.869344 17464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:08.870632 17464 hybrid_clock.cc:648] HybridClock initialized: now 1786008068870527 us; error 76 us; skew 500 ppm
I20260806 09:21:08.871064 17464 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:08 dist-test-slave-h3ft krb5kdc[17447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008068, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:08.874421 17464 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:21:08.876206 17464 webserver.cc:460] Webserver started at http://127.16.86.190:32935/ using document root <none> and password file <none>
I20260806 09:21:08.876955 17464 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:08.877069 17464 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:08.877352 17464 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:08.880740 17464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "f049d02690694b22af6a052a96381475"
format_stamp: "Formatted at 2026-08-06 09:21:08 on dist-test-slave-h3ft"
server_key: "7b66d045524a1b09b195391eb19bdb56"
server_key_iv: "79816029d9dd13c489caf81bfedbc437"
server_key_version: "encryptionkey@0"
I20260806 09:21:08.881469 17464 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "f049d02690694b22af6a052a96381475"
format_stamp: "Formatted at 2026-08-06 09:21:08 on dist-test-slave-h3ft"
server_key: "7b66d045524a1b09b195391eb19bdb56"
server_key_iv: "79816029d9dd13c489caf81bfedbc437"
server_key_version: "encryptionkey@0"
I20260806 09:21:08.887421 17464 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 09:21:08.891877 17488 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:08.893842 17464 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:21:08.894065 17464 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "f049d02690694b22af6a052a96381475"
format_stamp: "Formatted at 2026-08-06 09:21:08 on dist-test-slave-h3ft"
server_key: "7b66d045524a1b09b195391eb19bdb56"
server_key_iv: "79816029d9dd13c489caf81bfedbc437"
server_key_version: "encryptionkey@0"
I20260806 09:21:08.894290 17464 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:08.908483 17464 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:08.913270 17464 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:08.913595 17464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:08.936021 17464 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:35805
I20260806 09:21:08.936123 17561 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:35805 every 8 connection(s)
I20260806 09:21:08.937661 17464 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:08.939445 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 17464
I20260806 09:21:08.939792 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:08.940737 16730 external_mini_cluster.cc:1428] Setting key 514cfa6f786031239bbf13349bb1f17c
I20260806 09:21:08.944537 17562 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:21:08 dist-test-slave-h3ft krb5kdc[17447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:08.961752 17562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475: Bootstrap starting.
I20260806 09:21:08.966956 17562 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:08.968786 17562 log.cc:824] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:08.971704 17565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:08.943530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33738 (local address 127.16.86.190:35805)
0806 09:21:08.944457 (+ 927us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:08.944470 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:08.944530 (+ 60us) server_negotiation.cc:407] Connection header received
0806 09:21:08.946406 (+ 1876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:08.946433 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:08.946898 (+ 465us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:08.947376 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:08.949830 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:08.951505 (+ 1675us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:08.953575 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:08.954659 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:08.954885 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:08.960987 (+ 6102us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:08.960996 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:08.961018 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:08.961070 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:08.965608 (+ 4538us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:08.966626 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:08.966638 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:08.966653 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:08.966976 (+ 323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:08.967497 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:08.967505 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:08.967511 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:08.968429 (+ 918us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:08.968837 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:08.969479 (+ 642us) server_negotiation.cc:299] Negotiation successful
0806 09:21:08.970091 (+ 612us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":687,"thread_start_us":433,"threads_started":1}
I20260806 09:21:08.973356 17562 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475: No bootstrap required, opened a new log
I20260806 09:21:08.982674 17562 raft_consensus.cc:348] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f049d02690694b22af6a052a96381475" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35805 } }
I20260806 09:21:08.983287 17562 raft_consensus.cc:374] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:08.983385 17562 raft_consensus.cc:729] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f049d02690694b22af6a052a96381475, State: Initialized, Role: FOLLOWER
I20260806 09:21:08.984123 17562 consensus_queue.cc:260] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f049d02690694b22af6a052a96381475" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35805 } }
I20260806 09:21:08.984395 17562 raft_consensus.cc:388] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:08.984520 17562 raft_consensus.cc:482] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:08.984675 17562 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:08.987308 17562 raft_consensus.cc:504] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f049d02690694b22af6a052a96381475" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35805 } }
I20260806 09:21:08.988097 17562 leader_election.cc:302] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f049d02690694b22af6a052a96381475; no voters:
I20260806 09:21:08.988898 17562 leader_election.cc:288] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:08.989111 17567 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:08.989977 17567 raft_consensus.cc:686] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [term 1 LEADER]: Becoming Leader. State: Replica: f049d02690694b22af6a052a96381475, State: Running, Role: LEADER
I20260806 09:21:08.990720 17567 consensus_queue.cc:237] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f049d02690694b22af6a052a96381475" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35805 } }
I20260806 09:21:08.991716 17562 sys_catalog.cc:564] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:08.994344 17568 sys_catalog.cc:455] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f049d02690694b22af6a052a96381475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f049d02690694b22af6a052a96381475" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35805 } } }
I20260806 09:21:08.994949 17569 sys_catalog.cc:455] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f049d02690694b22af6a052a96381475. Latest consensus state: current_term: 1 leader_uuid: "f049d02690694b22af6a052a96381475" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f049d02690694b22af6a052a96381475" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35805 } } }
I20260806 09:21:08.995301 17568 sys_catalog.cc:458] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:08.995388 17569 sys_catalog.cc:458] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:08.998695 17575 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:09.006212 17575 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:09.018383 17575 catalog_manager.cc:1347] Generated new cluster ID: 5d11a35988674fe0865b60248684fc60
I20260806 09:21:09.018541 17575 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:09.031698 17575 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:09.034106 17575 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:09.042436 17575 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f049d02690694b22af6a052a96381475: Generated new TSK 0
I20260806 09:21:09.043689 17575 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:21:09.116294 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:35805
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:09.474790 17595 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:09.475204 17595 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:09.475371 17595 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:09.482914 17595 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:09.483115 17595 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:09.483294 17595 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:21:09.493654 17595 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:35805
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:09.495061 17595 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:09.496443 17595 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:09.511759 17607 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:09.511876 17605 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:09.511900 17604 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:09.512945 17595 server_base.cc:1034] running on GCE node
I20260806 09:21:09.513669 17595 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:09.514751 17595 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:09.516003 17595 hybrid_clock.cc:648] HybridClock initialized: now 1786008069515936 us; error 85 us; skew 500 ppm
I20260806 09:21:09.516389 17595 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:09 dist-test-slave-h3ft krb5kdc[17447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008069, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:09.520198 17595 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:21:09.521860 17595 webserver.cc:460] Webserver started at http://127.16.86.129:42733/ using document root <none> and password file <none>
I20260806 09:21:09.522588 17595 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:09.522733 17595 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:09.523138 17595 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:09.526144 17595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "149f3c7782fb42b8accd6c6cb2364984"
format_stamp: "Formatted at 2026-08-06 09:21:09 on dist-test-slave-h3ft"
server_key: "53a42dacf673dd1dd1aa61ccd8e26c9f"
server_key_iv: "475b7365355d15161d9660fc3b3b96c8"
server_key_version: "encryptionkey@0"
I20260806 09:21:09.526973 17595 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "149f3c7782fb42b8accd6c6cb2364984"
format_stamp: "Formatted at 2026-08-06 09:21:09 on dist-test-slave-h3ft"
server_key: "53a42dacf673dd1dd1aa61ccd8e26c9f"
server_key_iv: "475b7365355d15161d9660fc3b3b96c8"
server_key_version: "encryptionkey@0"
I20260806 09:21:09.532919 17595 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 09:21:09.537149 17614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:09.538902 17595 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 09:21:09.539223 17595 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "149f3c7782fb42b8accd6c6cb2364984"
format_stamp: "Formatted at 2026-08-06 09:21:09 on dist-test-slave-h3ft"
server_key: "53a42dacf673dd1dd1aa61ccd8e26c9f"
server_key_iv: "475b7365355d15161d9660fc3b3b96c8"
server_key_version: "encryptionkey@0"
I20260806 09:21:09.539462 17595 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:09.562630 17595 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:09.567287 17595 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:09.567593 17595 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:09.568717 17595 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:09.570200 17595 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:09.570313 17595 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:09.570473 17595 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:09.570545 17595 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:09.609140 17595 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:39977
I20260806 09:21:09.609261 17762 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:39977 every 8 connection(s)
I20260806 09:21:09.610523 17595 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:21:09.613811 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 17595
I20260806 09:21:09.614094 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:21:09.614950 16730 external_mini_cluster.cc:1428] Setting key 798e0786dc59f737fb804be6f2c846b5
Aug 06 09:21:09 dist-test-slave-h3ft krb5kdc[17447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008069, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:09.633710 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:09.613945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:40621 (local address 127.16.86.190:35805)
0806 09:21:09.614601 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:09.614610 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:09.615610 (+ 1000us) server_negotiation.cc:407] Connection header received
0806 09:21:09.617809 (+ 2199us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:09.617817 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:09.617909 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:09.618025 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:09.619619 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:09.620358 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:09.621911 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:09.622318 (+ 407us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:09.622459 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:09.626048 (+ 3589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:09.626053 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:09.626056 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:09.626088 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:09.629214 (+ 3126us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:09.630707 (+ 1493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:09.630711 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:09.630714 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:09.630854 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:09.631535 (+ 681us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:09.631539 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:09.631541 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:09.631857 (+ 316us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:09.631985 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:09.633229 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0806 09:21:09.633424 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":282,"threads_started":1}
I20260806 09:21:09.635959 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:09.614493 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35805 (local address 127.16.86.129:40621)
0806 09:21:09.615399 (+ 906us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:09.615458 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:09.617508 (+ 2050us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:09.618268 (+ 760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:09.618276 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:09.618816 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:09.619452 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:09.619469 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:09.620526 (+ 1057us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:09.620530 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:09.621770 (+ 1240us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:09.621782 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:09.622556 (+ 774us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:09.622564 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:09.622805 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:09.623551 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:09.623573 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:09.625833 (+ 2260us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:09.629466 (+ 3633us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:09.629484 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:09.629513 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:09.630515 (+ 1002us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:09.630988 (+ 473us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:09.630997 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:09.631032 (+ 35us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:09.631380 (+ 348us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:09.632017 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:09.632031 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:09.632695 (+ 664us) client_negotiation.cc:769] Sending connection context
0806 09:21:09.633106 (+ 411us) client_negotiation.cc:240] Negotiation successful
0806 09:21:09.633898 (+ 792us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":614,"thread_start_us":372,"threads_started":1}
I20260806 09:21:09.637866 17763 heartbeater.cc:347] Connected to a master server at 127.16.86.190:35805
I20260806 09:21:09.638358 17763 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:09.639734 17763 heartbeater.cc:511] Master 127.16.86.190:35805 requested a full tablet report, sending...
I20260806 09:21:09.642675 17512 ts_manager.cc:194] Registered new tserver with Master: 149f3c7782fb42b8accd6c6cb2364984 (127.16.86.129:39977)
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
I20260806 09:21:09.644501 17512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:40621
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:21:09.678951 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:35805
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:10.032526 17770 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:10.032867 17770 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:10.033036 17770 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:10.040417 17770 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:10.040542 17770 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:10.040719 17770 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:21:10.050966 17770 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:35805
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:10.052384 17770 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:10.053853 17770 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:10.068324 17775 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:10.068665 17778 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:10.069059 17776 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:10.072904 17770 server_base.cc:1034] running on GCE node
I20260806 09:21:10.073678 17770 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:10.074815 17770 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:10.076115 17770 hybrid_clock.cc:648] HybridClock initialized: now 1786008070076058 us; error 50 us; skew 500 ppm
I20260806 09:21:10.076496 17770 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:10 dist-test-slave-h3ft krb5kdc[17447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:10.080253 17770 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:21:10.082152 17770 webserver.cc:460] Webserver started at http://127.16.86.130:37691/ using document root <none> and password file <none>
I20260806 09:21:10.082882 17770 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:10.082993 17770 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:10.083369 17770 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:10.086418 17770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "fad1c372300b48cf86711cb7e2e8b287"
format_stamp: "Formatted at 2026-08-06 09:21:10 on dist-test-slave-h3ft"
server_key: "25fd8404fc4ebbfd444cda2424af4a04"
server_key_iv: "cd6dc059183261b2725e0e4f7c856026"
server_key_version: "encryptionkey@0"
I20260806 09:21:10.087276 17770 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "fad1c372300b48cf86711cb7e2e8b287"
format_stamp: "Formatted at 2026-08-06 09:21:10 on dist-test-slave-h3ft"
server_key: "25fd8404fc4ebbfd444cda2424af4a04"
server_key_iv: "cd6dc059183261b2725e0e4f7c856026"
server_key_version: "encryptionkey@0"
I20260806 09:21:10.093350 17770 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 09:21:10.097790 17788 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:10.100173 17770 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 09:21:10.100361 17770 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "fad1c372300b48cf86711cb7e2e8b287"
format_stamp: "Formatted at 2026-08-06 09:21:10 on dist-test-slave-h3ft"
server_key: "25fd8404fc4ebbfd444cda2424af4a04"
server_key_iv: "cd6dc059183261b2725e0e4f7c856026"
server_key_version: "encryptionkey@0"
I20260806 09:21:10.100571 17770 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:10.124178 17770 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:10.128697 17770 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:10.128985 17770 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:10.130085 17770 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:10.131664 17770 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:10.131752 17770 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:10.131843 17770 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:10.131893 17770 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:10.171773 17770 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:40195
I20260806 09:21:10.171914 17936 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:40195 every 8 connection(s)
I20260806 09:21:10.173255 17770 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:21:10.176254 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 17770
I20260806 09:21:10.176542 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:21:10.177250 16730 external_mini_cluster.cc:1428] Setting key 0fd7ae2ed66491d76e66f00e0e85602e
Aug 06 09:21:10 dist-test-slave-h3ft krb5kdc[17447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:10.199524 17939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:10.176652 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:44829 (local address 127.16.86.190:35805)
0806 09:21:10.177335 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:10.177343 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:10.178401 (+ 1058us) server_negotiation.cc:407] Connection header received
0806 09:21:10.180471 (+ 2070us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:10.180477 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:10.180579 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:10.180738 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:10.182503 (+ 1765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.183797 (+ 1294us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.185590 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.186439 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.186606 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:10.190903 (+ 4297us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:10.190913 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:10.190922 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:10.190979 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:10.195433 (+ 4454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:10.196307 (+ 874us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:10.196316 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:10.196323 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:10.196556 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:10.196996 (+ 440us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:10.197004 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:10.197010 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:10.197469 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:10.197678 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:10.198546 (+ 868us) server_negotiation.cc:299] Negotiation successful
0806 09:21:10.198872 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":331,"threads_started":1}
I20260806 09:21:10.200511 17940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:10.177221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35805 (local address 127.16.86.130:44829)
0806 09:21:10.178212 (+ 991us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:10.178261 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:10.180170 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:10.180972 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:10.180980 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:10.181575 (+ 595us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:10.182306 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:10.182327 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.184009 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.184013 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:10.185421 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:10.185435 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.186720 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.186726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:10.186940 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:10.187858 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:10.187885 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:10.190618 (+ 2733us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:10.195620 (+ 5002us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:10.195628 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:10.195642 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:10.196157 (+ 515us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:10.196679 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:10.196682 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:10.196685 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:10.196879 (+ 194us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:10.197661 (+ 782us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:10.197669 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:10.198038 (+ 369us) client_negotiation.cc:769] Sending connection context
0806 09:21:10.198364 (+ 326us) client_negotiation.cc:240] Negotiation successful
0806 09:21:10.198981 (+ 617us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":429,"threads_started":1}
I20260806 09:21:10.202147 17937 heartbeater.cc:347] Connected to a master server at 127.16.86.190:35805
I20260806 09:21:10.202488 17937 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:10.203306 17937 heartbeater.cc:511] Master 127.16.86.190:35805 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
I20260806 09:21:10.204914 17512 ts_manager.cc:194] Registered new tserver with Master: fad1c372300b48cf86711cb7e2e8b287 (127.16.86.130:40195)
I20260806 09:21:10.206054 17512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:44829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:21:10.240658 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:35805
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33319
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:10.648095 17943 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
I20260806 09:21:10.648412 17763 heartbeater.cc:503] Master 127.16.86.190:35805 was elected leader, sending a full tablet report...
W20260806 09:21:10.648535 17943 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:10.648722 17943 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:10.657585 17943 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:10.657819 17943 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:10.658039 17943 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:21:10.670228 17943 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33319
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:35805
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:10.671938 17943 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:10.673887 17943 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:10.689471 17949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:10.689631 17950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:10.689601 17952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:10.691109 17943 server_base.cc:1034] running on GCE node
I20260806 09:21:10.692000 17943 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:10.693255 17943 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:10.694523 17943 hybrid_clock.cc:648] HybridClock initialized: now 1786008070694471 us; error 47 us; skew 500 ppm
I20260806 09:21:10.694906 17943 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:10 dist-test-slave-h3ft krb5kdc[17447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:10.699065 17943 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:21:10.701161 17943 webserver.cc:460] Webserver started at http://127.16.86.131:37675/ using document root <none> and password file <none>
I20260806 09:21:10.701956 17943 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:10.702083 17943 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:10.702419 17943 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:10.706112 17943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "6a2461a7550b462c96c888a9a80b758e"
format_stamp: "Formatted at 2026-08-06 09:21:10 on dist-test-slave-h3ft"
server_key: "fc43e79cbaea4b853b6d54d7501dbeb4"
server_key_iv: "ba61a4a007e9937a513415762f2bb022"
server_key_version: "encryptionkey@0"
I20260806 09:21:10.706974 17943 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "6a2461a7550b462c96c888a9a80b758e"
format_stamp: "Formatted at 2026-08-06 09:21:10 on dist-test-slave-h3ft"
server_key: "fc43e79cbaea4b853b6d54d7501dbeb4"
server_key_iv: "ba61a4a007e9937a513415762f2bb022"
server_key_version: "encryptionkey@0"
I20260806 09:21:10.713624 17943 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:21:10.718364 17962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:10.720335 17943 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:21:10.720533 17943 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "6a2461a7550b462c96c888a9a80b758e"
format_stamp: "Formatted at 2026-08-06 09:21:10 on dist-test-slave-h3ft"
server_key: "fc43e79cbaea4b853b6d54d7501dbeb4"
server_key_iv: "ba61a4a007e9937a513415762f2bb022"
server_key_version: "encryptionkey@0"
I20260806 09:21:10.720762 17943 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:10.738783 17943 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:10.743727 17943 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:10.743999 17943 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:10.745311 17943 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:10.746950 17943 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:10.747108 17943 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260806 09:21:10.747308 17943 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:10.747386 17943 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:10.789498 17943 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:45391
I20260806 09:21:10.789532 18112 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:45391 every 8 connection(s)
I20260806 09:21:10.791137 17943 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:21:10.796909 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 17943
I20260806 09:21:10.797125 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:21:10.797616 16730 external_mini_cluster.cc:1428] Setting key d669cdb690c061af11477efd7a37949e
Aug 06 09:21:10 dist-test-slave-h3ft krb5kdc[17447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008070, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:10.817785 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:10.794719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:48787 (local address 127.16.86.190:35805)
0806 09:21:10.795500 (+ 781us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:10.795509 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:10.796493 (+ 984us) server_negotiation.cc:407] Connection header received
0806 09:21:10.799613 (+ 3120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:10.799617 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:10.799793 (+ 176us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:10.799936 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:10.802276 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.803189 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.805381 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.805885 (+ 504us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.805991 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:10.810569 (+ 4578us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:10.810574 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:10.810578 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:10.810616 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:10.814048 (+ 3432us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:10.815220 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:10.815225 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:10.815227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:10.815378 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:10.815883 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:10.815886 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:10.815888 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:10.816187 (+ 299us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:10.816313 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:10.817270 (+ 957us) server_negotiation.cc:299] Negotiation successful
0806 09:21:10.817460 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":430,"threads_started":1}
I20260806 09:21:10.819340 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:10.795190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35805 (local address 127.16.86.131:48787)
0806 09:21:10.796235 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:10.796301 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:10.799159 (+ 2858us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:10.800315 (+ 1156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:10.800333 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:10.801103 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:10.802074 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:10.802100 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.803452 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.803457 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:10.805207 (+ 1750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:10.805225 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.806158 (+ 933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.806164 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:10.806429 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:10.807406 (+ 977us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:10.807438 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:10.810297 (+ 2859us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:10.814250 (+ 3953us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:10.814261 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:10.814282 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:10.815102 (+ 820us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:10.815500 (+ 398us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:10.815504 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:10.815507 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:10.815764 (+ 257us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:10.816342 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:10.816351 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:10.816848 (+ 497us) client_negotiation.cc:769] Sending connection context
0806 09:21:10.817163 (+ 315us) client_negotiation.cc:240] Negotiation successful
0806 09:21:10.817735 (+ 572us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":722,"thread_start_us":389,"threads_started":1}
I20260806 09:21:10.821081 18113 heartbeater.cc:347] Connected to a master server at 127.16.86.190:35805
I20260806 09:21:10.821557 18113 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:10.822655 18113 heartbeater.cc:511] Master 127.16.86.190:35805 requested a full tablet report, sending...
I20260806 09:21:10.824456 17512 ts_manager.cc:194] Registered new tserver with Master: 6a2461a7550b462c96c888a9a80b758e (127.16.86.131:45391)
I20260806 09:21:10.825443 17512 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:48787
I20260806 09:21:10.827955 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:21:10.855460 18115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:10.837066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33754 (local address 127.16.86.190:35805)
0806 09:21:10.837419 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:10.837429 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:10.837831 (+ 402us) server_negotiation.cc:407] Connection header received
0806 09:21:10.838090 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:10.838101 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:10.838200 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:10.838331 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:10.840308 (+ 1977us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.841186 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.842624 (+ 1438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.843161 (+ 537us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.843337 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:10.846557 (+ 3220us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:10.846563 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:10.846567 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:10.846614 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:10.851045 (+ 4431us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:10.852363 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:10.852387 (+ 24us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:10.852394 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:10.852622 (+ 228us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:10.853402 (+ 780us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:10.853407 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:10.853410 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:10.853884 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:10.854136 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:10.854742 (+ 606us) server_negotiation.cc:299] Negotiation successful
0806 09:21:10.855152 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20260806 09:21:10.865720 17512 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:33754:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:21:10.873168 17512 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:21:10.915217 18128 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:10.889033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:45391 (local address 127.0.0.1:34362)
0806 09:21:10.894125 (+ 5092us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:10.894203 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:10.894512 (+ 309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:10.905499 (+ 10987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:10.905508 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:10.905550 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:10.906251 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:10.906268 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.908957 (+ 2689us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.908964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:10.912421 (+ 3457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:10.912430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.913976 (+ 1546us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.913982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:10.914186 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:10.914257 (+ 71us) client_negotiation.cc:769] Sending connection context
0806 09:21:10.914412 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 09:21:10.914634 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4683,"mutex_wait_us":186,"thread_start_us":4465,"threads_started":1}
I20260806 09:21:10.915336 18133 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:10.897546 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:40195 (local address 127.0.0.1:47618)
0806 09:21:10.905047 (+ 7501us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:10.905072 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:10.905247 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:10.906821 (+ 1574us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:10.906826 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:10.906852 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:10.907242 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:10.907251 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.908756 (+ 1505us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.908760 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:10.912383 (+ 3623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:10.912410 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.914448 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.914453 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:10.914617 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:10.914646 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 09:21:10.914736 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 09:21:10.914885 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7374,"mutex_wait_us":190,"thread_start_us":7224,"threads_started":1}
I20260806 09:21:10.916445 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:10.895040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47618 (local address 127.16.86.130:40195)
0806 09:21:10.905828 (+ 10788us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:10.905838 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:10.905868 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:21:10.905980 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:10.905991 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:10.906309 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:10.906551 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:10.907412 (+ 861us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.908550 (+ 1138us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.912634 (+ 4084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.914197 (+ 1563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.914376 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:10.914664 (+ 288us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:10.914815 (+ 151us) server_negotiation.cc:299] Negotiation successful
0806 09:21:10.916029 (+ 1214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10605,"thread_start_us":10475,"threads_started":1}
I20260806 09:21:10.916545 18131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:10.889881 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39977 (local address 127.0.0.1:34254)
0806 09:21:10.901867 (+ 11986us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:10.901893 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:10.902090 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:10.905837 (+ 3747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:10.905843 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:10.905872 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:10.906579 (+ 707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:10.906606 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.908957 (+ 2351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.908964 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:10.912178 (+ 3214us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:10.912195 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.915574 (+ 3379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.915580 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:10.915781 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:10.915819 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 09:21:10.915916 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 09:21:10.916123 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11784,"mutex_wait_us":14,"thread_start_us":11584,"threads_started":1}
I20260806 09:21:10.916584 18129 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:10.889410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34362 (local address 127.16.86.131:45391)
0806 09:21:10.904690 (+ 15280us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:10.904700 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:10.904736 (+ 36us) server_negotiation.cc:407] Connection header received
0806 09:21:10.904858 (+ 122us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:10.904868 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:10.905131 (+ 263us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:10.906483 (+ 1352us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:10.906842 (+ 359us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.908747 (+ 1905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.912670 (+ 3923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.913751 (+ 1081us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.913922 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:10.914192 (+ 270us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:10.916005 (+ 1813us) server_negotiation.cc:299] Negotiation successful
0806 09:21:10.916218 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15078,"thread_start_us":14866,"threads_started":1}
I20260806 09:21:10.917384 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:10.889980 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34254 (local address 127.16.86.129:39977)
0806 09:21:10.904093 (+ 14113us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:10.904109 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:10.904155 (+ 46us) server_negotiation.cc:407] Connection header received
0806 09:21:10.904317 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:10.904327 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:10.904701 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:10.904986 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:10.906913 (+ 1927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.908754 (+ 1841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.912370 (+ 3616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:10.914902 (+ 2532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:10.915157 (+ 255us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:10.915409 (+ 252us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:10.916814 (+ 1405us) server_negotiation.cc:299] Negotiation successful
0806 09:21:10.917033 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13882,"thread_start_us":13664,"threads_started":1}
I20260806 09:21:10.920974 18032 tablet_service.cc:1467] Processing CreateTablet for tablet 533e617365734d62bc830bcdb6fc591f (DEFAULT_TABLE table=test-table [id=39ec62078d454ec89b4b72b003daefa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:10.921623 17681 tablet_service.cc:1467] Processing CreateTablet for tablet 533e617365734d62bc830bcdb6fc591f (DEFAULT_TABLE table=test-table [id=39ec62078d454ec89b4b72b003daefa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:10.921066 17859 tablet_service.cc:1467] Processing CreateTablet for tablet 533e617365734d62bc830bcdb6fc591f (DEFAULT_TABLE table=test-table [id=39ec62078d454ec89b4b72b003daefa5]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:10.922986 18032 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533e617365734d62bc830bcdb6fc591f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:10.923251 17681 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533e617365734d62bc830bcdb6fc591f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:10.923439 17859 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 533e617365734d62bc830bcdb6fc591f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:10.948870 18137 tablet_bootstrap.cc:492] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e: Bootstrap starting.
I20260806 09:21:10.956148 18137 tablet_bootstrap.cc:654] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:10.956208 18135 tablet_bootstrap.cc:492] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287: Bootstrap starting.
I20260806 09:21:10.957803 18137 log.cc:824] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:10.959517 18134 tablet_bootstrap.cc:492] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984: Bootstrap starting.
I20260806 09:21:10.961059 18135 tablet_bootstrap.cc:654] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:10.962206 18135 log.cc:824] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:10.963333 18134 tablet_bootstrap.cc:654] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:10.964836 18134 log.cc:824] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:10.965999 18137 tablet_bootstrap.cc:492] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e: No bootstrap required, opened a new log
I20260806 09:21:10.966504 18137 ts_tablet_manager.cc:1397] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e: Time spent bootstrapping tablet: real 0.018s user 0.009s sys 0.000s
I20260806 09:21:10.974200 18135 tablet_bootstrap.cc:492] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287: No bootstrap required, opened a new log
I20260806 09:21:10.974646 18135 ts_tablet_manager.cc:1397] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287: Time spent bootstrapping tablet: real 0.019s user 0.013s sys 0.005s
I20260806 09:21:10.978714 18137 raft_consensus.cc:348] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2461a7550b462c96c888a9a80b758e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 45391 } } peers { permanent_uuid: "fad1c372300b48cf86711cb7e2e8b287" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 40195 } } peers { permanent_uuid: "149f3c7782fb42b8accd6c6cb2364984" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39977 } }
I20260806 09:21:10.979312 18137 raft_consensus.cc:374] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:10.979470 18137 raft_consensus.cc:729] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6a2461a7550b462c96c888a9a80b758e, State: Initialized, Role: FOLLOWER
I20260806 09:21:10.980206 18137 consensus_queue.cc:260] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2461a7550b462c96c888a9a80b758e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 45391 } } peers { permanent_uuid: "fad1c372300b48cf86711cb7e2e8b287" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 40195 } } peers { permanent_uuid: "149f3c7782fb42b8accd6c6cb2364984" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39977 } }
I20260806 09:21:10.981148 18134 tablet_bootstrap.cc:492] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984: No bootstrap required, opened a new log
I20260806 09:21:10.981595 18134 ts_tablet_manager.cc:1397] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984: Time spent bootstrapping tablet: real 0.022s user 0.014s sys 0.008s
I20260806 09:21:10.981666 18113 heartbeater.cc:503] Master 127.16.86.190:35805 was elected leader, sending a full tablet report...
I20260806 09:21:10.982327 18137 ts_tablet_manager.cc:1428] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e: Time spent starting tablet: real 0.016s user 0.018s sys 0.000s
I20260806 09:21:10.987102 18135 raft_consensus.cc:348] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2461a7550b462c96c888a9a80b758e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 45391 } } peers { permanent_uuid: "fad1c372300b48cf86711cb7e2e8b287" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 40195 } } peers { permanent_uuid: "149f3c7782fb42b8accd6c6cb2364984" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39977 } }
I20260806 09:21:10.987617 18135 raft_consensus.cc:374] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:10.987758 18135 raft_consensus.cc:729] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fad1c372300b48cf86711cb7e2e8b287, State: Initialized, Role: FOLLOWER
I20260806 09:21:10.988474 18135 consensus_queue.cc:260] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2461a7550b462c96c888a9a80b758e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 45391 } } peers { permanent_uuid: "fad1c372300b48cf86711cb7e2e8b287" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 40195 } } peers { permanent_uuid: "149f3c7782fb42b8accd6c6cb2364984" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39977 } }
I20260806 09:21:10.990533 17937 heartbeater.cc:503] Master 127.16.86.190:35805 was elected leader, sending a full tablet report...
I20260806 09:21:10.991060 18135 ts_tablet_manager.cc:1428] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287: Time spent starting tablet: real 0.016s user 0.007s sys 0.007s
I20260806 09:21:10.991843 18134 raft_consensus.cc:348] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2461a7550b462c96c888a9a80b758e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 45391 } } peers { permanent_uuid: "fad1c372300b48cf86711cb7e2e8b287" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 40195 } } peers { permanent_uuid: "149f3c7782fb42b8accd6c6cb2364984" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39977 } }
I20260806 09:21:10.992372 18134 raft_consensus.cc:374] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:10.992507 18134 raft_consensus.cc:729] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 149f3c7782fb42b8accd6c6cb2364984, State: Initialized, Role: FOLLOWER
I20260806 09:21:10.993284 18134 consensus_queue.cc:260] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2461a7550b462c96c888a9a80b758e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 45391 } } peers { permanent_uuid: "fad1c372300b48cf86711cb7e2e8b287" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 40195 } } peers { permanent_uuid: "149f3c7782fb42b8accd6c6cb2364984" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39977 } }
I20260806 09:21:10.995505 18134 ts_tablet_manager.cc:1428] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984: Time spent starting tablet: real 0.014s user 0.010s sys 0.004s
W20260806 09:21:11.043159 18114 tablet.cc:2369] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:21:11.064164 18143 raft_consensus.cc:482] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:21:11.064574 18143 raft_consensus.cc:504] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2461a7550b462c96c888a9a80b758e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 45391 } } peers { permanent_uuid: "fad1c372300b48cf86711cb7e2e8b287" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 40195 } } peers { permanent_uuid: "149f3c7782fb42b8accd6c6cb2364984" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39977 } }
I20260806 09:21:11.066795 18143 leader_election.cc:288] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fad1c372300b48cf86711cb7e2e8b287 (127.16.86.130:40195), 149f3c7782fb42b8accd6c6cb2364984 (127.16.86.129:39977)
I20260806 09:21:11.074498 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.067360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:42421 (local address 127.16.86.130:40195)
0806 09:21:11.067975 (+ 615us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:11.067979 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:11.067994 (+ 15us) server_negotiation.cc:407] Connection header received
0806 09:21:11.068055 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:11.068059 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:11.068119 (+ 60us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:11.068222 (+ 103us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:11.068779 (+ 557us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.069919 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.072084 (+ 2165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.073409 (+ 1325us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.073547 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:11.073613 (+ 66us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:11.074043 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 09:21:11.074187 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455}
I20260806 09:21:11.074551 18116 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.067339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:40195 (local address 127.16.86.131:42421)
0806 09:21:11.067630 (+ 291us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:11.067654 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:11.067802 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:11.068252 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:11.068257 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:11.068286 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:11.068572 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:11.068589 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.070077 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.070083 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:11.071897 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:11.071911 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.073586 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.073593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:11.073821 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:11.073862 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:21:11.073982 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 09:21:11.074163 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":108}
I20260806 09:21:11.075791 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.067456 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:41389 (local address 127.16.86.129:39977)
0806 09:21:11.067780 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:11.067788 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:11.068473 (+ 685us) server_negotiation.cc:407] Connection header received
0806 09:21:11.068711 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:11.068719 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:11.068798 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:11.068877 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:11.069586 (+ 709us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.070653 (+ 1067us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.073290 (+ 2637us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.074558 (+ 1268us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.074750 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:11.074839 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:11.075427 (+ 588us) server_negotiation.cc:299] Negotiation successful
0806 09:21:11.075580 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260806 09:21:11.075906 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.067452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39977 (local address 127.16.86.131:41389)
0806 09:21:11.068330 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:11.068357 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:11.068505 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:11.068939 (+ 434us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:11.068943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:11.068971 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:11.069396 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:11.069409 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.070842 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.070846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:11.073076 (+ 2230us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:11.073093 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.075078 (+ 1985us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.075084 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:11.075223 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:11.075248 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:21:11.075332 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 09:21:11.075515 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":741,"thread_start_us":467,"threads_started":1}
I20260806 09:21:11.076071 17884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533e617365734d62bc830bcdb6fc591f" candidate_uuid: "6a2461a7550b462c96c888a9a80b758e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad1c372300b48cf86711cb7e2e8b287" is_pre_election: true
I20260806 09:21:11.076711 17884 raft_consensus.cc:2454] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2461a7550b462c96c888a9a80b758e in term 0.
I20260806 09:21:11.077226 17704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533e617365734d62bc830bcdb6fc591f" candidate_uuid: "6a2461a7550b462c96c888a9a80b758e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149f3c7782fb42b8accd6c6cb2364984" is_pre_election: true
I20260806 09:21:11.077704 17968 leader_election.cc:302] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6a2461a7550b462c96c888a9a80b758e, fad1c372300b48cf86711cb7e2e8b287; no voters:
I20260806 09:21:11.077872 17704 raft_consensus.cc:2454] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6a2461a7550b462c96c888a9a80b758e in term 0.
I20260806 09:21:11.078214 18143 raft_consensus.cc:2781] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:21:11.078474 18143 raft_consensus.cc:482] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:21:11.078676 18143 raft_consensus.cc:3037] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:11.081517 18143 raft_consensus.cc:504] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2461a7550b462c96c888a9a80b758e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 45391 } } peers { permanent_uuid: "fad1c372300b48cf86711cb7e2e8b287" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 40195 } } peers { permanent_uuid: "149f3c7782fb42b8accd6c6cb2364984" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39977 } }
I20260806 09:21:11.082298 18143 leader_election.cc:288] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [CANDIDATE]: Term 1 election: Requested vote from peers fad1c372300b48cf86711cb7e2e8b287 (127.16.86.130:40195), 149f3c7782fb42b8accd6c6cb2364984 (127.16.86.129:39977)
I20260806 09:21:11.082973 17704 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533e617365734d62bc830bcdb6fc591f" candidate_uuid: "6a2461a7550b462c96c888a9a80b758e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "149f3c7782fb42b8accd6c6cb2364984"
I20260806 09:21:11.083128 17884 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "533e617365734d62bc830bcdb6fc591f" candidate_uuid: "6a2461a7550b462c96c888a9a80b758e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fad1c372300b48cf86711cb7e2e8b287"
I20260806 09:21:11.083320 17704 raft_consensus.cc:3037] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:11.083410 17884 raft_consensus.cc:3037] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:11.086001 17704 raft_consensus.cc:2454] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2461a7550b462c96c888a9a80b758e in term 1.
I20260806 09:21:11.086247 17884 raft_consensus.cc:2454] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6a2461a7550b462c96c888a9a80b758e in term 1.
I20260806 09:21:11.086663 17966 leader_election.cc:302] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 149f3c7782fb42b8accd6c6cb2364984, 6a2461a7550b462c96c888a9a80b758e; no voters:
I20260806 09:21:11.087141 18143 raft_consensus.cc:2781] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:11.087782 18143 raft_consensus.cc:686] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [term 1 LEADER]: Becoming Leader. State: Replica: 6a2461a7550b462c96c888a9a80b758e, State: Running, Role: LEADER
I20260806 09:21:11.088444 18143 consensus_queue.cc:237] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2461a7550b462c96c888a9a80b758e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 45391 } } peers { permanent_uuid: "fad1c372300b48cf86711cb7e2e8b287" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 40195 } } peers { permanent_uuid: "149f3c7782fb42b8accd6c6cb2364984" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39977 } }
I20260806 09:21:11.094590 17511 catalog_manager.cc:5495] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e reported cstate change: term changed from 0 to 1, leader changed from <none> to 6a2461a7550b462c96c888a9a80b758e (127.16.86.131). New cstate: current_term: 1 leader_uuid: "6a2461a7550b462c96c888a9a80b758e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6a2461a7550b462c96c888a9a80b758e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 45391 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fad1c372300b48cf86711cb7e2e8b287" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 40195 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "149f3c7782fb42b8accd6c6cb2364984" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39977 } health_report { overall_health: UNKNOWN } } }
W20260806 09:21:11.113519 17765 tablet.cc:2369] T 533e617365734d62bc830bcdb6fc591f P 149f3c7782fb42b8accd6c6cb2364984: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 09:21:11 dist-test-slave-h3ft krb5kdc[17447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:21:11.122288 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.105369 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34264 (local address 127.16.86.129:39977)
0806 09:21:11.105673 (+ 304us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:11.105680 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:11.105997 (+ 317us) server_negotiation.cc:407] Connection header received
0806 09:21:11.106231 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:11.106243 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:11.106345 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:11.106451 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:11.108392 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.109415 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.110807 (+ 1392us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.111385 (+ 578us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.111502 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:11.116126 (+ 4624us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:11.116138 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:11.116154 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:11.116198 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:11.119329 (+ 3131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:11.120070 (+ 741us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:11.120077 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:11.120080 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:11.120250 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:11.120650 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:11.120654 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:11.120656 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:11.121026 (+ 370us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:11.121289 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:11.121673 (+ 384us) server_negotiation.cc:299] Negotiation successful
0806 09:21:11.121865 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
Aug 06 09:21:11 dist-test-slave-h3ft krb5kdc[17447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
W20260806 09:21:11.175868 17938 tablet.cc:2369] T 533e617365734d62bc830bcdb6fc591f P fad1c372300b48cf86711cb7e2e8b287: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 09:21:11 dist-test-slave-h3ft krb5kdc[17447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008071, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:21:11.191532 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.175884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34268 (local address 127.16.86.129:39977)
0806 09:21:11.176742 (+ 858us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:11.176747 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:11.177292 (+ 545us) server_negotiation.cc:407] Connection header received
0806 09:21:11.177523 (+ 231us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:11.177533 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:11.177599 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:11.177706 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:11.179231 (+ 1525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.180003 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.181307 (+ 1304us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.181710 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.181854 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:11.186041 (+ 4187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:11.186046 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:11.186049 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:11.186083 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:11.188852 (+ 2769us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:11.189485 (+ 633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:11.189491 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:11.189499 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:11.189629 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:11.190053 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:11.190059 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:11.190062 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:11.190335 (+ 273us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:11.190621 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:11.190900 (+ 279us) server_negotiation.cc:299] Negotiation successful
0806 09:21:11.191188 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":738}
Aug 06 09:21:11 dist-test-slave-h3ft krb5kdc[17447](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:21:11 dist-test-slave-h3ft krb5kdc[17447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:21:11.252648 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.235277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34278 (local address 127.16.86.129:39977)
0806 09:21:11.235577 (+ 300us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:11.235584 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:11.235932 (+ 348us) server_negotiation.cc:407] Connection header received
0806 09:21:11.236132 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:11.236136 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:11.236225 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:11.236345 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:11.238061 (+ 1716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.238907 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.240416 (+ 1509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.240908 (+ 492us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.241024 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:11.246258 (+ 5234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:11.246265 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:11.246269 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:11.246334 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:11.249519 (+ 3185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:11.250438 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:11.250443 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:11.250445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:11.250598 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:11.251131 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:11.251135 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:11.251138 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:11.251530 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:11.251763 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:11.252103 (+ 340us) server_negotiation.cc:299] Negotiation successful
0806 09:21:11.252321 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
W20260806 09:21:11.255172 17649 tablet_service.cc:2908] Rejecting scan request for tablet 533e617365734d62bc830bcdb6fc591f: Uninitialized: clean time has not yet been initialized
I20260806 09:21:11.260165 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 17595
I20260806 09:21:11.274838 18130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.261540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34290 (local address 127.16.86.129:39977)
0806 09:21:11.261789 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:11.261794 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:11.262106 (+ 312us) server_negotiation.cc:407] Connection header received
0806 09:21:11.262269 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:11.262273 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:11.262388 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:11.262492 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:11.263820 (+ 1328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.264852 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.266046 (+ 1194us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.266498 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.266620 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:11.269141 (+ 2521us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:11.269146 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:11.269150 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:11.269188 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:11.272152 (+ 2964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:11.272842 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:11.272848 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:11.272851 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:11.273016 (+ 165us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:11.273411 (+ 395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:11.273415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:11.273417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:11.273794 (+ 377us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:11.274070 (+ 276us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:11.274307 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 09:21:11.274546 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 09:21:11.275695 17757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:11.372604 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 17595
I20260806 09:21:11.401986 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 17770
Aug 06 09:21:11 dist-test-slave-h3ft krb5kdc[17447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:21:11.419777 18132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.403256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47626 (local address 127.16.86.130:40195)
0806 09:21:11.403469 (+ 213us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:11.403474 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:11.403497 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:21:11.403615 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:11.403620 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:11.403698 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:11.403795 (+ 97us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:11.405436 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.406374 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.407855 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.408341 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.408462 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:11.413068 (+ 4606us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:11.413089 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:11.413113 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:11.413171 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:11.416584 (+ 3413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:11.417536 (+ 952us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:11.417545 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:11.417548 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:11.417705 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:11.418093 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:11.418096 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:11.418098 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:11.418565 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:11.418849 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:11.419213 (+ 364us) server_negotiation.cc:299] Negotiation successful
0806 09:21:11.419442 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:21:11.420665 17932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:11.513265 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 17770
I20260806 09:21:11.518468 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.517673 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39977 (local address 127.16.86.131:50763)
0806 09:21:11.517989 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:11.518230 (+ 241us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
W20260806 09:21:11.518918 17966 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39977: connect: Connection refused (error 111)
I20260806 09:21:11.524644 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.523899 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39977 (local address 127.16.86.131:47047)
0806 09:21:11.524182 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:11.524345 (+ 163us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39977: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
W20260806 09:21:11.525054 17966 consensus_peers.cc:487] T 533e617365734d62bc830bcdb6fc591f P 6a2461a7550b462c96c888a9a80b758e -> Peer 149f3c7782fb42b8accd6c6cb2364984 (127.16.86.129:39977): Couldn't send request to peer 149f3c7782fb42b8accd6c6cb2364984. Status: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39977: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 09:21:11.543710 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 17943
Aug 06 09:21:11 dist-test-slave-h3ft krb5kdc[17447](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008071, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:21:11.561987 18168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.544691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34368 (local address 127.16.86.131:45391)
0806 09:21:11.545328 (+ 637us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:11.545333 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:11.545359 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:21:11.545436 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:11.545440 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:11.545521 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:11.545639 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:11.547065 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.547851 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.549963 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:11.550416 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:11.550527 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:11.555560 (+ 5033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:11.555579 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:11.555600 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:11.555645 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:11.558868 (+ 3223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:11.559724 (+ 856us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:11.559732 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:11.559734 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:11.559914 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:11.560323 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:11.560326 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:11.560334 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:11.560730 (+ 396us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:11.560989 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:11.561299 (+ 310us) server_negotiation.cc:299] Negotiation successful
0806 09:21:11.561522 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":504,"thread_start_us":336,"threads_started":1}
I20260806 09:21:11.562930 18107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:11.667601 18147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:11.666987 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:40195 (local address 127.16.86.131:49115)
0806 09:21:11.667226 (+ 239us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:11.667355 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.130:40195: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20260806 09:21:11.667595 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 17943
I20260806 09:21:11.696452 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 17464
I20260806 09:21:11.697877 17551 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:11.814567 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 17464
2026-08-06T09:21:11Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4975 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:21:11 dist-test-slave-h3ft krb5kdc[18174](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:21:11 dist-test-slave-h3ft krb5kdc[18174](info): set up 2 sockets
Aug 06 09:21:11 dist-test-slave-h3ft krb5kdc[18174](info): commencing operation
krb5kdc: starting...
W20260806 09:21:13.153944 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.283s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:21:13 dist-test-slave-h3ft krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:21:13Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:13Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:21:13.412719 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41457
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:41457
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:13.779645 18193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:13.779976 18193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:13.780121 18193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:13.787670 18193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:13.787806 18193 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 09:21:13.787871 18193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:13.787981 18193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:13.788057 18193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:13.788151 18193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:21:13.798697 18193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:41457
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41457
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:13.800206 18193 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:13.801673 18193 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:13.815176 18199 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:13.815426 18204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:13.815487 18201 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:13.816646 18193 server_base.cc:1034] running on GCE node
I20260806 09:21:13.817744 18193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:13.819787 18193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:13.821108 18193 hybrid_clock.cc:648] HybridClock initialized: now 1786008073820975 us; error 76 us; skew 500 ppm
I20260806 09:21:13.821630 18193 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:13 dist-test-slave-h3ft krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008073, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:13.831079 18193 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:21:13.834017 18193 webserver.cc:460] Webserver started at http://127.16.86.190:41793/ using document root <none> and password file <none>
I20260806 09:21:13.835139 18193 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:13.835296 18193 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:13.835740 18193 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:13.840036 18193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "46e32d7285844709a162bde458a4f1bb"
format_stamp: "Formatted at 2026-08-06 09:21:13 on dist-test-slave-h3ft"
server_key: "7380544db8199333917c596c4415fab0"
server_key_iv: "6ff06f0caae1fde0569b10f8c13fdebd"
server_key_version: "encryptionkey@0"
I20260806 09:21:13.841141 18193 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "46e32d7285844709a162bde458a4f1bb"
format_stamp: "Formatted at 2026-08-06 09:21:13 on dist-test-slave-h3ft"
server_key: "7380544db8199333917c596c4415fab0"
server_key_iv: "6ff06f0caae1fde0569b10f8c13fdebd"
server_key_version: "encryptionkey@0"
I20260806 09:21:13.848729 18193 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260806 09:21:13.853715 18213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:13.855729 18193 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:21:13.855995 18193 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "46e32d7285844709a162bde458a4f1bb"
format_stamp: "Formatted at 2026-08-06 09:21:13 on dist-test-slave-h3ft"
server_key: "7380544db8199333917c596c4415fab0"
server_key_iv: "6ff06f0caae1fde0569b10f8c13fdebd"
server_key_version: "encryptionkey@0"
I20260806 09:21:13.856302 18193 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:13.876495 18193 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:13.882441 18193 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:13.882822 18193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:13.913446 18193 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:41457
I20260806 09:21:13.913513 18301 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:41457 every 8 connection(s)
I20260806 09:21:13.915470 18193 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:13.920068 18302 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:13.921191 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 18193
I20260806 09:21:13.921468 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:13.922370 16730 external_mini_cluster.cc:1428] Setting key 59aa7e679233b919bb5673466e3fd09a
I20260806 09:21:13.936782 18302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb: Bootstrap starting.
Aug 06 09:21:13 dist-test-slave-h3ft krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008073, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:13.941229 18302 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:13.942384 18302 log.cc:824] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:13.946274 18302 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb: No bootstrap required, opened a new log
I20260806 09:21:13.951973 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:13.925172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45586 (local address 127.16.86.190:41457)
0806 09:21:13.926363 (+ 1191us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:13.926380 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:13.926440 (+ 60us) server_negotiation.cc:407] Connection header received
0806 09:21:13.928526 (+ 2086us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:13.928560 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:13.929118 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:13.929557 (+ 439us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:13.931537 (+ 1980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:13.933093 (+ 1556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:13.935176 (+ 2083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:13.936023 (+ 847us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:13.936192 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:13.942053 (+ 5861us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:13.942064 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:13.942091 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:13.942159 (+ 68us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:13.946480 (+ 4321us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:13.947525 (+ 1045us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:13.947544 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:13.947552 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:13.947760 (+ 208us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:13.948197 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:13.948202 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:13.948204 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:13.948967 (+ 763us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:13.949400 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:13.949882 (+ 482us) server_negotiation.cc:299] Negotiation successful
0806 09:21:13.950483 (+ 601us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":890,"thread_start_us":577,"threads_started":1}
I20260806 09:21:13.955231 18302 raft_consensus.cc:348] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e32d7285844709a162bde458a4f1bb" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41457 } }
I20260806 09:21:13.955752 18302 raft_consensus.cc:374] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:13.955879 18302 raft_consensus.cc:729] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46e32d7285844709a162bde458a4f1bb, State: Initialized, Role: FOLLOWER
I20260806 09:21:13.956485 18302 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e32d7285844709a162bde458a4f1bb" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41457 } }
I20260806 09:21:13.956748 18302 raft_consensus.cc:388] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:13.956867 18302 raft_consensus.cc:482] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:13.956991 18302 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:13.959821 18302 raft_consensus.cc:504] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e32d7285844709a162bde458a4f1bb" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41457 } }
I20260806 09:21:13.960397 18302 leader_election.cc:302] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46e32d7285844709a162bde458a4f1bb; no voters:
I20260806 09:21:13.961086 18302 leader_election.cc:288] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:13.961347 18309 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:13.962241 18309 raft_consensus.cc:686] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [term 1 LEADER]: Becoming Leader. State: Replica: 46e32d7285844709a162bde458a4f1bb, State: Running, Role: LEADER
I20260806 09:21:13.963521 18309 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e32d7285844709a162bde458a4f1bb" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41457 } }
I20260806 09:21:13.964471 18302 sys_catalog.cc:564] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:13.967201 18311 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46e32d7285844709a162bde458a4f1bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e32d7285844709a162bde458a4f1bb" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41457 } } }
I20260806 09:21:13.967823 18311 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:13.967147 18310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46e32d7285844709a162bde458a4f1bb. Latest consensus state: current_term: 1 leader_uuid: "46e32d7285844709a162bde458a4f1bb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46e32d7285844709a162bde458a4f1bb" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41457 } } }
I20260806 09:21:13.967998 18310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:13.971057 18314 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:13.977579 18314 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:13.988991 18314 catalog_manager.cc:1347] Generated new cluster ID: 2385fdaa132f4475baf0d726059b7c99
I20260806 09:21:13.989221 18314 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:14.004492 18314 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:14.006471 18314 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:14.016428 18314 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb: Generated new TSK 0
I20260806 09:21:14.017611 18314 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:21:14.083247 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41457
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:14.515558 18334 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:14.516067 18334 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:14.516243 18334 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:14.524717 18334 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:14.524864 18334 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:14.524961 18334 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:21:14.525060 18334 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:21:14.536224 18334 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.86.190:41457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:14.537909 18334 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:14.539608 18334 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:14.556906 18343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:14.557040 18344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:14.556964 18347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:14.560959 18334 server_base.cc:1034] running on GCE node
I20260806 09:21:14.561868 18334 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:14.563171 18334 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:14.564482 18334 hybrid_clock.cc:648] HybridClock initialized: now 1786008074564315 us; error 106 us; skew 500 ppm
I20260806 09:21:14.564936 18334 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:14 dist-test-slave-h3ft krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008074, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:14.569109 18334 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:21:14.571291 18334 webserver.cc:460] Webserver started at http://127.16.86.129:36667/ using document root <none> and password file <none>
I20260806 09:21:14.572297 18334 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:14.572443 18334 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:14.572866 18334 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:14.576505 18334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "e0542bd8775b4312a6ac4205a3123e11"
format_stamp: "Formatted at 2026-08-06 09:21:14 on dist-test-slave-h3ft"
server_key: "affbb734e4aabb1c906e476ad964b8e4"
server_key_iv: "2842cbb1deda8d79b25b1f22a6894f06"
server_key_version: "encryptionkey@0"
I20260806 09:21:14.577510 18334 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "e0542bd8775b4312a6ac4205a3123e11"
format_stamp: "Formatted at 2026-08-06 09:21:14 on dist-test-slave-h3ft"
server_key: "affbb734e4aabb1c906e476ad964b8e4"
server_key_iv: "2842cbb1deda8d79b25b1f22a6894f06"
server_key_version: "encryptionkey@0"
I20260806 09:21:14.584760 18334 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260806 09:21:14.589442 18357 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:14.591550 18334 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260806 09:21:14.591815 18334 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "e0542bd8775b4312a6ac4205a3123e11"
format_stamp: "Formatted at 2026-08-06 09:21:14 on dist-test-slave-h3ft"
server_key: "affbb734e4aabb1c906e476ad964b8e4"
server_key_iv: "2842cbb1deda8d79b25b1f22a6894f06"
server_key_version: "encryptionkey@0"
I20260806 09:21:14.592132 18334 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:14.618973 18334 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:14.624843 18334 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:14.625213 18334 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:14.629225 18334 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:14.629369 18334 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:14.629541 18334 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:14.629640 18334 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:21:14 dist-test-slave-h3ft krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008074, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:14.659509 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:14.629935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:48447 (local address 127.16.86.190:41457)
0806 09:21:14.630791 (+ 856us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:14.630796 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:14.632671 (+ 1875us) server_negotiation.cc:407] Connection header received
0806 09:21:14.635516 (+ 2845us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:14.635524 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:14.635639 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:14.635778 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:14.638760 (+ 2982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:14.640036 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:14.643180 (+ 3144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:14.644373 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:14.644649 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:14.650703 (+ 6054us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:14.650718 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:14.650728 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:14.650809 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:14.655584 (+ 4775us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:14.656871 (+ 1287us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:14.656877 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:14.656881 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:14.657031 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:14.657529 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:14.657534 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:14.657538 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:14.657843 (+ 305us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:14.658023 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:14.658829 (+ 806us) server_negotiation.cc:299] Negotiation successful
0806 09:21:14.659140 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":736,"thread_start_us":632,"threads_started":1}
I20260806 09:21:14.660678 18371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:14.630963 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41457 (local address 127.16.86.129:48447)
0806 09:21:14.632409 (+ 1446us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:14.632467 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:14.635179 (+ 2712us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:14.636132 (+ 953us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:14.636151 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:14.637294 (+ 1143us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:14.638442 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:14.638485 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:14.640242 (+ 1757us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:14.640253 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:14.642855 (+ 2602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:14.642869 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:14.644742 (+ 1873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:14.644748 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:14.645046 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:14.645941 (+ 895us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:14.645987 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:14.650250 (+ 4263us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:14.655835 (+ 5585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:14.655853 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:14.655880 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:14.656731 (+ 851us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:14.657152 (+ 421us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:14.657160 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:14.657164 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:14.657382 (+ 218us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:14.657981 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:14.657990 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:14.658420 (+ 430us) client_negotiation.cc:769] Sending connection context
0806 09:21:14.658712 (+ 292us) client_negotiation.cc:240] Negotiation successful
0806 09:21:14.659333 (+ 621us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1010,"thread_start_us":530,"threads_started":1}
I20260806 09:21:14.684906 18457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:14.669268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41457 (local address 127.16.86.129:60593)
0806 09:21:14.670072 (+ 804us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:14.670091 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:14.670285 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:14.670620 (+ 335us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:14.670624 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:14.671271 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:14.671856 (+ 585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:14.671873 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:14.673320 (+ 1447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:14.673331 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:14.675038 (+ 1707us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:14.675055 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:14.676111 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:14.676117 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:14.676307 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:14.676997 (+ 690us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:14.677015 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:14.677772 (+ 757us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:14.681373 (+ 3601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:14.681383 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:14.681387 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:14.682111 (+ 724us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:14.682497 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:14.682502 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:14.682506 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:14.682658 (+ 152us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:14.683353 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:14.683358 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:14.683571 (+ 213us) client_negotiation.cc:769] Sending connection context
0806 09:21:14.683753 (+ 182us) client_negotiation.cc:240] Negotiation successful
0806 09:21:14.684235 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":460,"threads_started":1}
I20260806 09:21:14.685072 18370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:14.669174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:60593 (local address 127.16.86.190:41457)
0806 09:21:14.669397 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:14.669403 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:14.670197 (+ 794us) server_negotiation.cc:407] Connection header received
0806 09:21:14.670421 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:14.670426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:14.670487 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:14.670586 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:14.672061 (+ 1475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:14.673062 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:14.675209 (+ 2147us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:14.675887 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:14.676038 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:14.677965 (+ 1927us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:14.677971 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:14.677981 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:14.678012 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:14.681234 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:14.682235 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:14.682243 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:14.682248 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:14.682376 (+ 128us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:14.682776 (+ 400us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:14.682784 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:14.682789 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:14.683242 (+ 453us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:14.683408 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:14.684475 (+ 1067us) server_negotiation.cc:299] Negotiation successful
0806 09:21:14.684718 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 09:21:14.700212 18334 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:41115
I20260806 09:21:14.700289 18523 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:41115 every 8 connection(s)
I20260806 09:21:14.701944 18334 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:21:14.704953 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 18334
I20260806 09:21:14.705049 18524 heartbeater.cc:347] Connected to a master server at 127.16.86.190:41457
I20260806 09:21:14.705235 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:21:14.705408 18524 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:14.706117 16730 external_mini_cluster.cc:1428] Setting key 85d19d1ece809136ba446d40f34e92ce
I20260806 09:21:14.706266 18524 heartbeater.cc:511] Master 127.16.86.190:41457 requested a full tablet report, sending...
I20260806 09:21:14.709245 18231 ts_manager.cc:194] Registered new tserver with Master: e0542bd8775b4312a6ac4205a3123e11 (127.16.86.129:41115)
I20260806 09:21:14.711882 18231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:48447
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:21:14.764322 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41457
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:15.182562 18532 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:15.182958 18532 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:15.183166 18532 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:15.191260 18532 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:15.191417 18532 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:15.191597 18532 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:21:15.191735 18532 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:21:15.203428 18532 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.86.190:41457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:15.205251 18532 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:15.206957 18532 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:15.224133 18543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:15.224274 18541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:15.224591 18539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:15.225639 18532 server_base.cc:1034] running on GCE node
I20260806 09:21:15.226495 18532 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:15.227813 18532 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:15.229092 18532 hybrid_clock.cc:648] HybridClock initialized: now 1786008075228991 us; error 68 us; skew 500 ppm
I20260806 09:21:15.229485 18532 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:15 dist-test-slave-h3ft krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008075, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:15.233472 18532 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:21:15.235525 18532 webserver.cc:460] Webserver started at http://127.16.86.130:33965/ using document root <none> and password file <none>
I20260806 09:21:15.236475 18532 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:15.236616 18532 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:15.237021 18532 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:15.240777 18532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "7d3a8b99c9c743dea16530c9ee41ae68"
format_stamp: "Formatted at 2026-08-06 09:21:15 on dist-test-slave-h3ft"
server_key: "2526fcba805bfae98aaf777dd81efde4"
server_key_iv: "dece4c23283ceb4b18df29db9049ecb6"
server_key_version: "encryptionkey@0"
I20260806 09:21:15.241899 18532 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "7d3a8b99c9c743dea16530c9ee41ae68"
format_stamp: "Formatted at 2026-08-06 09:21:15 on dist-test-slave-h3ft"
server_key: "2526fcba805bfae98aaf777dd81efde4"
server_key_iv: "dece4c23283ceb4b18df29db9049ecb6"
server_key_version: "encryptionkey@0"
I20260806 09:21:15.251223 18532 fs_manager.cc:691] Time spent creating directory manager: real 0.009s user 0.010s sys 0.001s
I20260806 09:21:15.256521 18552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:15.258458 18532 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260806 09:21:15.258711 18532 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "7d3a8b99c9c743dea16530c9ee41ae68"
format_stamp: "Formatted at 2026-08-06 09:21:15 on dist-test-slave-h3ft"
server_key: "2526fcba805bfae98aaf777dd81efde4"
server_key_iv: "dece4c23283ceb4b18df29db9049ecb6"
server_key_version: "encryptionkey@0"
I20260806 09:21:15.259104 18532 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:15.277812 18532 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:15.283205 18532 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:15.283555 18532 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:15.287097 18532 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:15.287225 18532 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:15.287401 18532 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:15.287497 18532 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:21:15 dist-test-slave-h3ft krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008075, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:15.319983 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:15.287774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:50333 (local address 127.16.86.190:41457)
0806 09:21:15.288700 (+ 926us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:15.288708 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:15.290077 (+ 1369us) server_negotiation.cc:407] Connection header received
0806 09:21:15.293765 (+ 3688us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:15.293773 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:15.293906 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:15.294086 (+ 180us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:15.297747 (+ 3661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.299099 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.302265 (+ 3166us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.302922 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.303104 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:15.309848 (+ 6744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:15.309857 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:15.309862 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:15.309909 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:15.313898 (+ 3989us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:15.315201 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:15.315211 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:15.315214 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:15.315390 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:15.316114 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:15.316125 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:15.316131 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:15.316772 (+ 641us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:15.317036 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:15.319233 (+ 2197us) server_negotiation.cc:299] Negotiation successful
0806 09:21:15.319543 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":761,"thread_start_us":605,"threads_started":1}
I20260806 09:21:15.321245 18566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:15.288660 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41457 (local address 127.16.86.130:50333)
0806 09:21:15.289803 (+ 1143us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:15.289884 (+ 81us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:15.293281 (+ 3397us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:15.294527 (+ 1246us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:15.294540 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:15.295969 (+ 1429us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:15.297436 (+ 1467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:15.297483 (+ 47us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.299335 (+ 1852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.299343 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:15.301952 (+ 2609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:15.302005 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.303601 (+ 1596us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.303614 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:15.303987 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:15.305357 (+ 1370us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:15.305389 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:15.309569 (+ 4180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:15.314074 (+ 4505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:15.314085 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:15.314105 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:15.314966 (+ 861us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:15.315524 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:15.315529 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:15.315533 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:15.315851 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:15.317169 (+ 1318us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:15.317182 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:15.318117 (+ 935us) client_negotiation.cc:769] Sending connection context
0806 09:21:15.318638 (+ 521us) client_negotiation.cc:240] Negotiation successful
0806 09:21:15.319552 (+ 914us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":757,"thread_start_us":473,"threads_started":1}
I20260806 09:21:15.343988 18532 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:33429
I20260806 09:21:15.344099 18716 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:33429 every 8 connection(s)
I20260806 09:21:15.345652 18532 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:21:15.348608 18717 heartbeater.cc:347] Connected to a master server at 127.16.86.190:41457
I20260806 09:21:15.348631 18679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:15.329659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41457 (local address 127.16.86.130:46809)
0806 09:21:15.330896 (+ 1237us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:15.330931 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:15.331205 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:15.331925 (+ 720us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:15.331935 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:15.332732 (+ 797us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:15.333168 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:15.333180 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.335103 (+ 1923us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.335112 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:15.337003 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:15.337016 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.338345 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.338350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:15.338498 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:15.339488 (+ 990us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:15.339519 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:15.341207 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:15.345251 (+ 4044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:15.345256 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:15.345261 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:15.345841 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:15.346242 (+ 401us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:15.346248 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:15.346252 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:15.346411 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:15.347346 (+ 935us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:15.347352 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:15.347594 (+ 242us) client_negotiation.cc:769] Sending connection context
0806 09:21:15.347735 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 09:21:15.348113 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1006,"thread_start_us":774,"threads_started":1}
I20260806 09:21:15.348934 18717 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:15.349509 18565 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:15.329037 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:46809 (local address 127.16.86.190:41457)
0806 09:21:15.329349 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:15.329360 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:15.331076 (+ 1716us) server_negotiation.cc:407] Connection header received
0806 09:21:15.331444 (+ 368us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:15.331455 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:15.331557 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:15.331623 (+ 66us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:15.333377 (+ 1754us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.334798 (+ 1421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.337191 (+ 2393us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.337867 (+ 676us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.338045 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:15.341538 (+ 3493us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:15.341545 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:15.341548 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:15.341635 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:15.345068 (+ 3433us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:15.345974 (+ 906us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:15.345979 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:15.345981 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:15.346114 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:15.346548 (+ 434us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:15.346554 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:15.346557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:15.347166 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:15.348773 (+ 1607us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:15.348959 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 09:21:15.349257 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:21:15.349889 18717 heartbeater.cc:511] Master 127.16.86.190:41457 requested a full tablet report, sending...
I20260806 09:21:15.351544 18231 ts_manager.cc:194] Registered new tserver with Master: 7d3a8b99c9c743dea16530c9ee41ae68 (127.16.86.130:33429)
I20260806 09:21:15.352491 18231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:50333
I20260806 09:21:15.355973 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 18532
I20260806 09:21:15.356199 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:21:15.356714 16730 external_mini_cluster.cc:1428] Setting key 0f0cd690aa71d0c3a0855d57f234d7ce
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:21:15.416947 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41457
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:43867
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:21:15.715700 18524 heartbeater.cc:503] Master 127.16.86.190:41457 was elected leader, sending a full tablet report...
W20260806 09:21:15.832942 18721 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:15.833312 18721 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:15.833482 18721 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:15.840943 18721 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:15.841111 18721 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:15.841267 18721 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:21:15.841415 18721 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:21:15.852036 18721 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:43867
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.86.190:41457
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:15.853508 18721 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:15.855060 18721 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:15.871320 18729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:15.871425 18733 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:15.871358 18727 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:15.872870 18721 server_base.cc:1034] running on GCE node
I20260806 09:21:15.873754 18721 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:15.874897 18721 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:15.876220 18721 hybrid_clock.cc:648] HybridClock initialized: now 1786008075876121 us; error 71 us; skew 500 ppm
I20260806 09:21:15.876616 18721 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:15 dist-test-slave-h3ft krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008075, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:15.880571 18721 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:21:15.882445 18721 webserver.cc:460] Webserver started at http://127.16.86.131:43149/ using document root <none> and password file <none>
I20260806 09:21:15.883368 18721 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:15.883495 18721 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:15.883872 18721 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:15.887363 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "5b42eb25377842c2bec71feb76b51de3"
format_stamp: "Formatted at 2026-08-06 09:21:15 on dist-test-slave-h3ft"
server_key: "77110f6a0adcf205281b7c378cb786f9"
server_key_iv: "ad45ecb4aaf0fdb30e43ff460d29a3a7"
server_key_version: "encryptionkey@0"
I20260806 09:21:15.888262 18721 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "5b42eb25377842c2bec71feb76b51de3"
format_stamp: "Formatted at 2026-08-06 09:21:15 on dist-test-slave-h3ft"
server_key: "77110f6a0adcf205281b7c378cb786f9"
server_key_iv: "ad45ecb4aaf0fdb30e43ff460d29a3a7"
server_key_version: "encryptionkey@0"
I20260806 09:21:15.894582 18721 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 09:21:15.899202 18742 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:15.900936 18721 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 09:21:15.901144 18721 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "5b42eb25377842c2bec71feb76b51de3"
format_stamp: "Formatted at 2026-08-06 09:21:15 on dist-test-slave-h3ft"
server_key: "77110f6a0adcf205281b7c378cb786f9"
server_key_iv: "ad45ecb4aaf0fdb30e43ff460d29a3a7"
server_key_version: "encryptionkey@0"
I20260806 09:21:15.901386 18721 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:15.917050 18721 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:15.921854 18721 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:15.922158 18721 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:15.925484 18721 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:15.925590 18721 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:15.925760 18721 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:15.925851 18721 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:21:15 dist-test-slave-h3ft krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008075, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:15.960060 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:15.926690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:50375 (local address 127.16.86.190:41457)
0806 09:21:15.927404 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:15.927413 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:15.929066 (+ 1653us) server_negotiation.cc:407] Connection header received
0806 09:21:15.932237 (+ 3171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:15.932248 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:15.932403 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:15.932610 (+ 207us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:15.936308 (+ 3698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.937583 (+ 1275us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.941698 (+ 4115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.942254 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.942434 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:15.950048 (+ 7614us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:15.950064 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:15.950074 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:15.950141 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:15.954702 (+ 4561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:15.955862 (+ 1160us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:15.955869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:15.955875 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:15.956052 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:15.956841 (+ 789us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:15.956853 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:15.956860 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:15.957467 (+ 607us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:15.957718 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:15.958933 (+ 1215us) server_negotiation.cc:299] Negotiation successful
0806 09:21:15.959371 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":540,"thread_start_us":360,"threads_started":1}
I20260806 09:21:15.961525 18754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:15.927574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41457 (local address 127.16.86.131:50375)
0806 09:21:15.928805 (+ 1231us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:15.928870 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:15.931771 (+ 2901us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:15.933081 (+ 1310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:15.933100 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:15.934467 (+ 1367us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:15.935985 (+ 1518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:15.936036 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.937848 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.937855 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:15.941431 (+ 3576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:15.941483 (+ 52us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.942674 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.942686 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:15.943225 (+ 539us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:15.944488 (+ 1263us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:15.944546 (+ 58us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:15.949553 (+ 5007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:15.954888 (+ 5335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:15.954900 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:15.954926 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:15.955663 (+ 737us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:15.956179 (+ 516us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:15.956184 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:15.956187 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:15.956587 (+ 400us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:15.957728 (+ 1141us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:15.957737 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:15.958336 (+ 599us) client_negotiation.cc:769] Sending connection context
0806 09:21:15.958715 (+ 379us) client_negotiation.cc:240] Negotiation successful
0806 09:21:15.959502 (+ 787us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":851,"thread_start_us":525,"threads_started":1}
I20260806 09:21:15.979241 18721 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:36907
I20260806 09:21:15.979257 18905 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:36907 every 8 connection(s)
I20260806 09:21:15.981413 18721 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:21:15.984076 18906 heartbeater.cc:347] Connected to a master server at 127.16.86.190:41457
I20260806 09:21:15.984542 18906 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:15.985571 18906 heartbeater.cc:511] Master 127.16.86.190:41457 requested a full tablet report, sending...
I20260806 09:21:15.985882 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 18721
I20260806 09:21:15.986096 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:21:15.986644 16730 external_mini_cluster.cc:1428] Setting key 5d3b254020f6d82f0231561da69dacd3
I20260806 09:21:15.987658 18231 ts_manager.cc:194] Registered new tserver with Master: 5b42eb25377842c2bec71feb76b51de3 (127.16.86.131:36907)
I20260806 09:21:15.988693 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:15.969922 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:38563 (local address 127.16.86.190:41457)
0806 09:21:15.970204 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:15.970215 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:15.971197 (+ 982us) server_negotiation.cc:407] Connection header received
0806 09:21:15.971393 (+ 196us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:15.971403 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:15.971525 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:15.971690 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:15.973134 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.974653 (+ 1519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.977043 (+ 2390us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.977868 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.978051 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:15.981352 (+ 3301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:15.981359 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:15.981363 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:15.981400 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:15.985081 (+ 3681us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:15.986123 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:15.986134 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:15.986139 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:15.986334 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:15.986926 (+ 592us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:15.986933 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:15.986936 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:15.987407 (+ 471us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:15.987584 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:15.988010 (+ 426us) server_negotiation.cc:299] Negotiation successful
0806 09:21:15.988262 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 09:21:15.988716 18885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:15.969976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41457 (local address 127.16.86.131:38563)
0806 09:21:15.971047 (+ 1071us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:15.971072 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:15.971245 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:15.971739 (+ 494us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:15.971743 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:15.972295 (+ 552us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:15.972897 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:15.972913 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.974848 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.974854 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:15.976888 (+ 2034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:15.976901 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:15.978425 (+ 1524us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:15.978438 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:15.978729 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:15.979749 (+ 1020us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:15.979776 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:15.981087 (+ 1311us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:15.985224 (+ 4137us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:15.985229 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:15.985233 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:15.985932 (+ 699us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:15.986516 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:15.986522 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:15.986526 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:15.986748 (+ 222us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:15.987542 (+ 794us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:15.987547 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:15.987746 (+ 199us) client_negotiation.cc:769] Sending connection context
0806 09:21:15.987894 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 09:21:15.988258 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":858,"thread_start_us":698,"threads_started":1}
I20260806 09:21:15.989132 18231 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:50375
I20260806 09:21:15.991498 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:21:16 dist-test-slave-h3ft krb5kdc[18174](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:21:16 dist-test-slave-h3ft krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:16.057835 18753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.041779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45600 (local address 127.16.86.190:41457)
0806 09:21:16.041987 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:16.041991 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:16.042549 (+ 558us) server_negotiation.cc:407] Connection header received
0806 09:21:16.042764 (+ 215us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:16.042770 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:16.042847 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:16.042978 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:16.044776 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.045613 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.047068 (+ 1455us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.047489 (+ 421us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.047606 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.052202 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:16.052207 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:16.052211 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:16.052249 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:16.055224 (+ 2975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:16.055929 (+ 705us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:16.055933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:16.055935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:16.056057 (+ 122us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:16.056559 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:16.056563 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:16.056565 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:16.056856 (+ 291us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:16.057063 (+ 207us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:16.057379 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 09:21:16.057540 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:21:16.063627 18231 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:21:16.072290 18231 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:21:16.114548 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.088890 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47658 (local address 127.16.86.130:33429)
0806 09:21:16.100467 (+ 11577us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:16.100478 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:16.101881 (+ 1403us) server_negotiation.cc:407] Connection header received
0806 09:21:16.102036 (+ 155us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:16.102045 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:16.102393 (+ 348us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:16.102643 (+ 250us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:16.104101 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.106236 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.111548 (+ 5312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.112765 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.112947 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.113215 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:16.113946 (+ 731us) server_negotiation.cc:299] Negotiation successful
0806 09:21:16.114173 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11331,"thread_start_us":11132,"threads_started":1}
I20260806 09:21:16.114627 18922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.088750 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:33429 (local address 127.0.0.1:47658)
0806 09:21:16.101283 (+ 12533us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:16.101337 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:16.101615 (+ 278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:16.102968 (+ 1353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:16.102978 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:16.103075 (+ 97us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:16.103835 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.103854 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.106506 (+ 2652us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.106514 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.111284 (+ 4770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.111300 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.113156 (+ 1856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.113162 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.113444 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.113651 (+ 207us) client_negotiation.cc:769] Sending connection context
0806 09:21:16.113841 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 09:21:16.114106 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12362,"mutex_wait_us":87,"spinlock_wait_cycles":20608,"thread_start_us":12187,"threads_started":1}
I20260806 09:21:16.114681 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.089232 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54018 (local address 127.16.86.129:41115)
0806 09:21:16.099789 (+ 10557us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:16.099805 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:16.101888 (+ 2083us) server_negotiation.cc:407] Connection header received
0806 09:21:16.102040 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:16.102049 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:16.102330 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:16.102561 (+ 231us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:16.103490 (+ 929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.106242 (+ 2752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.111388 (+ 5146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.113170 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.113341 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.113626 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:16.114054 (+ 428us) server_negotiation.cc:299] Negotiation successful
0806 09:21:16.114326 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9659,"thread_start_us":9970,"threads_started":1}
I20260806 09:21:16.115701 18921 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.088687 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:41115 (local address 127.0.0.1:54018)
0806 09:21:16.101268 (+ 12581us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:16.101329 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:16.101615 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:16.102576 (+ 961us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:16.102584 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:16.102631 (+ 47us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:16.103203 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.103224 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.106798 (+ 3574us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.106807 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.111097 (+ 4290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.111115 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.113439 (+ 2324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.113446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.113755 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.113799 (+ 44us) client_negotiation.cc:769] Sending connection context
0806 09:21:16.114989 (+ 1190us) client_negotiation.cc:240] Negotiation successful
0806 09:21:16.115207 (+ 218us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12221,"mutex_wait_us":157,"spinlock_wait_cycles":15488,"thread_start_us":12028,"threads_started":1}
I20260806 09:21:16.119486 18624 tablet_service.cc:1467] Processing CreateTablet for tablet 04198ebbf1b745508a195318e541c43a (DEFAULT_TABLE table=test-table [id=1e1cf2019c8b40aeb43b569c1b347dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:16.119685 18431 tablet_service.cc:1467] Processing CreateTablet for tablet 04198ebbf1b745508a195318e541c43a (DEFAULT_TABLE table=test-table [id=1e1cf2019c8b40aeb43b569c1b347dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:16.121160 18624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04198ebbf1b745508a195318e541c43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:16.121665 18431 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04198ebbf1b745508a195318e541c43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:16.122089 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.102468 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45966 (local address 127.16.86.131:36907)
0806 09:21:16.108559 (+ 6091us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:16.108572 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:16.109161 (+ 589us) server_negotiation.cc:407] Connection header received
0806 09:21:16.109423 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:16.109433 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:16.109749 (+ 316us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:16.110010 (+ 261us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:16.110937 (+ 927us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.112758 (+ 1821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.119234 (+ 6476us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.120464 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.120627 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.120916 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:16.121516 (+ 600us) server_negotiation.cc:299] Negotiation successful
0806 09:21:16.121718 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5809,"thread_start_us":5547,"threads_started":1}
I20260806 09:21:16.122161 18927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.102961 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36907 (local address 127.0.0.1:45966)
0806 09:21:16.108981 (+ 6020us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:16.109011 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:16.109223 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:16.110014 (+ 791us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:16.110019 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:16.110047 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:16.110567 (+ 520us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.110580 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.112947 (+ 2367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.112954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.116008 (+ 3054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.116024 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.120969 (+ 4945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.120979 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.121202 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.121329 (+ 127us) client_negotiation.cc:769] Sending connection context
0806 09:21:16.121455 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 09:21:16.121625 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5835,"mutex_wait_us":1,"thread_start_us":5607,"threads_started":1}
I20260806 09:21:16.126274 18815 tablet_service.cc:1467] Processing CreateTablet for tablet 04198ebbf1b745508a195318e541c43a (DEFAULT_TABLE table=test-table [id=1e1cf2019c8b40aeb43b569c1b347dba]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:16.127983 18815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 04198ebbf1b745508a195318e541c43a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:16.137830 18928 tablet_bootstrap.cc:492] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11: Bootstrap starting.
I20260806 09:21:16.142760 18928 tablet_bootstrap.cc:654] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:16.142937 18932 tablet_bootstrap.cc:492] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3: Bootstrap starting.
I20260806 09:21:16.143054 18929 tablet_bootstrap.cc:492] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68: Bootstrap starting.
I20260806 09:21:16.146521 18928 log.cc:824] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:16.147298 18929 tablet_bootstrap.cc:654] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:16.147295 18932 tablet_bootstrap.cc:654] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:16.148721 18932 log.cc:824] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:16.148721 18929 log.cc:824] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:16.150633 18928 tablet_bootstrap.cc:492] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11: No bootstrap required, opened a new log
I20260806 09:21:16.151065 18928 ts_tablet_manager.cc:1397] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11: Time spent bootstrapping tablet: real 0.014s user 0.008s sys 0.000s
I20260806 09:21:16.152180 18929 tablet_bootstrap.cc:492] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68: No bootstrap required, opened a new log
I20260806 09:21:16.152432 18932 tablet_bootstrap.cc:492] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3: No bootstrap required, opened a new log
I20260806 09:21:16.152520 18929 ts_tablet_manager.cc:1397] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.007s
I20260806 09:21:16.152762 18932 ts_tablet_manager.cc:1397] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.004s
I20260806 09:21:16.160661 18929 raft_consensus.cc:348] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.161105 18929 raft_consensus.cc:374] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:16.161209 18929 raft_consensus.cc:729] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3a8b99c9c743dea16530c9ee41ae68, State: Initialized, Role: FOLLOWER
I20260806 09:21:16.161937 18929 consensus_queue.cc:260] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.163300 18717 heartbeater.cc:503] Master 127.16.86.190:41457 was elected leader, sending a full tablet report...
I20260806 09:21:16.163975 18929 ts_tablet_manager.cc:1428] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68: Time spent starting tablet: real 0.011s user 0.008s sys 0.005s
I20260806 09:21:16.164235 18932 raft_consensus.cc:348] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.164764 18932 raft_consensus.cc:374] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:16.164863 18932 raft_consensus.cc:729] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b42eb25377842c2bec71feb76b51de3, State: Initialized, Role: FOLLOWER
I20260806 09:21:16.165580 18932 consensus_queue.cc:260] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.166085 18928 raft_consensus.cc:348] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.166600 18928 raft_consensus.cc:374] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:16.166698 18928 raft_consensus.cc:729] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0542bd8775b4312a6ac4205a3123e11, State: Initialized, Role: FOLLOWER
I20260806 09:21:16.167145 18906 heartbeater.cc:503] Master 127.16.86.190:41457 was elected leader, sending a full tablet report...
I20260806 09:21:16.167493 18928 consensus_queue.cc:260] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.167891 18932 ts_tablet_manager.cc:1428] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3: Time spent starting tablet: real 0.015s user 0.012s sys 0.004s
I20260806 09:21:16.170897 18928 ts_tablet_manager.cc:1428] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11: Time spent starting tablet: real 0.020s user 0.009s sys 0.006s
W20260806 09:21:16.205513 18526 tablet.cc:2369] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:21:16.234027 18907 tablet.cc:2369] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:21:16.348855 18718 tablet.cc:2369] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:21:16.513171 18941 raft_consensus.cc:482] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:21:16.513680 18941 raft_consensus.cc:504] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.515915 18941 leader_election.cc:288] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7d3a8b99c9c743dea16530c9ee41ae68 (127.16.86.130:33429), 5b42eb25377842c2bec71feb76b51de3 (127.16.86.131:36907)
I20260806 09:21:16.526518 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.516513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:51465 (local address 127.16.86.130:33429)
0806 09:21:16.516741 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:16.516746 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:16.517556 (+ 810us) server_negotiation.cc:407] Connection header received
0806 09:21:16.517898 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:16.517907 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:16.518000 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:16.518081 (+ 81us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:16.519116 (+ 1035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.520319 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.523835 (+ 3516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.525251 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.525430 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.525504 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:16.526063 (+ 559us) server_negotiation.cc:299] Negotiation successful
0806 09:21:16.526207 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:21:16.526502 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.517698 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:40095 (local address 127.16.86.131:36907)
0806 09:21:16.517928 (+ 230us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:16.517937 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:16.518235 (+ 298us) server_negotiation.cc:407] Connection header received
0806 09:21:16.518415 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:16.518419 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:16.518488 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:16.518586 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:16.519358 (+ 772us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.520384 (+ 1026us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.523835 (+ 3451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.525188 (+ 1353us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.525422 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.525504 (+ 82us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:16.526055 (+ 551us) server_negotiation.cc:299] Negotiation successful
0806 09:21:16.526207 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 09:21:16.526772 18946 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.517397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36907 (local address 127.16.86.129:40095)
0806 09:21:16.518114 (+ 717us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:16.518136 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:16.518275 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:16.518633 (+ 358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:16.518639 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:16.518672 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:16.519196 (+ 524us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.519206 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.520530 (+ 1324us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.520533 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.523632 (+ 3099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.523649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.525596 (+ 1947us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.525601 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.525772 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.525854 (+ 82us) client_negotiation.cc:769] Sending connection context
0806 09:21:16.525979 (+ 125us) client_negotiation.cc:240] Negotiation successful
0806 09:21:16.526176 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"mutex_wait_us":209,"thread_start_us":355,"threads_started":1}
I20260806 09:21:16.526804 18944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.516478 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:33429 (local address 127.16.86.129:51465)
0806 09:21:16.517381 (+ 903us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:16.517407 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:16.517654 (+ 247us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:16.518327 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:16.518334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:16.518375 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:16.518856 (+ 481us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.518870 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.520527 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.520533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.523632 (+ 3099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.523649 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.525574 (+ 1925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.525584 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.525786 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.525927 (+ 141us) client_negotiation.cc:769] Sending connection context
0806 09:21:16.526010 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 09:21:16.526176 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":706,"mutex_wait_us":277,"thread_start_us":475,"threads_started":1}
I20260806 09:21:16.528131 18652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04198ebbf1b745508a195318e541c43a" candidate_uuid: "e0542bd8775b4312a6ac4205a3123e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" is_pre_election: true
I20260806 09:21:16.528227 18840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04198ebbf1b745508a195318e541c43a" candidate_uuid: "e0542bd8775b4312a6ac4205a3123e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b42eb25377842c2bec71feb76b51de3" is_pre_election: true
I20260806 09:21:16.528731 18652 raft_consensus.cc:2454] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0542bd8775b4312a6ac4205a3123e11 in term 0.
I20260806 09:21:16.528849 18840 raft_consensus.cc:2454] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e0542bd8775b4312a6ac4205a3123e11 in term 0.
I20260806 09:21:16.529588 18358 leader_election.cc:302] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d3a8b99c9c743dea16530c9ee41ae68, e0542bd8775b4312a6ac4205a3123e11; no voters:
I20260806 09:21:16.530138 18941 raft_consensus.cc:2781] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:21:16.530390 18941 raft_consensus.cc:482] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:21:16.530566 18941 raft_consensus.cc:3037] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:16.533793 18941 raft_consensus.cc:504] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.534543 18941 leader_election.cc:288] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [CANDIDATE]: Term 1 election: Requested vote from peers 7d3a8b99c9c743dea16530c9ee41ae68 (127.16.86.130:33429), 5b42eb25377842c2bec71feb76b51de3 (127.16.86.131:36907)
I20260806 09:21:16.535212 18652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04198ebbf1b745508a195318e541c43a" candidate_uuid: "e0542bd8775b4312a6ac4205a3123e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d3a8b99c9c743dea16530c9ee41ae68"
I20260806 09:21:16.535305 18840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "04198ebbf1b745508a195318e541c43a" candidate_uuid: "e0542bd8775b4312a6ac4205a3123e11" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b42eb25377842c2bec71feb76b51de3"
I20260806 09:21:16.535456 18652 raft_consensus.cc:3037] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:16.535627 18840 raft_consensus.cc:3037] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:16.538234 18652 raft_consensus.cc:2454] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0542bd8775b4312a6ac4205a3123e11 in term 1.
I20260806 09:21:16.538292 18840 raft_consensus.cc:2454] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e0542bd8775b4312a6ac4205a3123e11 in term 1.
I20260806 09:21:16.538810 18358 leader_election.cc:302] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7d3a8b99c9c743dea16530c9ee41ae68, e0542bd8775b4312a6ac4205a3123e11; no voters:
I20260806 09:21:16.539279 18941 raft_consensus.cc:2781] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:16.539922 18941 raft_consensus.cc:686] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 1 LEADER]: Becoming Leader. State: Replica: e0542bd8775b4312a6ac4205a3123e11, State: Running, Role: LEADER
I20260806 09:21:16.540653 18941 consensus_queue.cc:237] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.546456 18229 catalog_manager.cc:5495] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 reported cstate change: term changed from 0 to 1, leader changed from <none> to e0542bd8775b4312a6ac4205a3123e11 (127.16.86.129). New cstate: current_term: 1 leader_uuid: "e0542bd8775b4312a6ac4205a3123e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } health_report { overall_health: UNKNOWN } } }
I20260806 09:21:16.594812 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.587904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54024 (local address 127.16.86.129:41115)
0806 09:21:16.588140 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:16.588145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:16.588671 (+ 526us) server_negotiation.cc:407] Connection header received
0806 09:21:16.589093 (+ 422us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:16.589098 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:16.589218 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:16.589391 (+ 173us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:16.589994 (+ 603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.590795 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.592375 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.592830 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.592955 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.593556 (+ 601us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:16.594151 (+ 595us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:16.594268 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:16.594481 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 09:21:16.594563 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:21:16.605470 18840 raft_consensus.cc:1261] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3 [term 1 FOLLOWER]: Refusing update from remote peer e0542bd8775b4312a6ac4205a3123e11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:16.605484 18652 raft_consensus.cc:1261] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68 [term 1 FOLLOWER]: Refusing update from remote peer e0542bd8775b4312a6ac4205a3123e11: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:16.606679 18941 consensus_queue.cc:1035] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:16.607106 18947 consensus_queue.cc:1035] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:16.625926 18953 mvcc.cc:205] Tried to move back new op lower bound from 7315489081762967552 to 7315489081515315200. Current Snapshot: MvccSnapshot[applied={T|T < 7315489081762967552}]
I20260806 09:21:16.626641 18955 mvcc.cc:205] Tried to move back new op lower bound from 7315489081762967552 to 7315489081515315200. Current Snapshot: MvccSnapshot[applied={T|T < 7315489081762967552}]
I20260806 09:21:16.651503 18229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:16.651970 18229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:16.656395 18229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb: Sending DeleteTablet for 3 replicas of tablet 04198ebbf1b745508a195318e541c43a
I20260806 09:21:16.658269 18815 tablet_service.cc:1514] Processing DeleteTablet for tablet 04198ebbf1b745508a195318e541c43a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:16 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:45966
I20260806 09:21:16.658278 18431 tablet_service.cc:1514] Processing DeleteTablet for tablet 04198ebbf1b745508a195318e541c43a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:16 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:54018
I20260806 09:21:16.659715 18624 tablet_service.cc:1514] Processing DeleteTablet for tablet 04198ebbf1b745508a195318e541c43a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:16 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:47658
I20260806 09:21:16.660272 18229 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45600:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:21:16.661258 18229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:21:16.664777 18971 tablet_replica.cc:330] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68: stopping tablet replica
I20260806 09:21:16.665683 18971 raft_consensus.cc:2229] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:16.666249 18971 raft_consensus.cc:2258] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:16.667131 18967 tablet_replica.cc:330] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3: stopping tablet replica
I20260806 09:21:16.667593 18967 raft_consensus.cc:2229] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:16.668112 18967 raft_consensus.cc:2258] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:16.671969 18430 tablet_service.cc:1467] Processing CreateTablet for tablet 038f402314264aebacc175ac7fa56bb2 (DEFAULT_TABLE table=test-table [id=4c95b262ec1a4dbdbf740517878966b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:16.672650 18430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038f402314264aebacc175ac7fa56bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:16.673091 18971 ts_tablet_manager.cc:1905] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:16.673461 18967 ts_tablet_manager.cc:1905] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:16.677315 18624 tablet_service.cc:1467] Processing CreateTablet for tablet 038f402314264aebacc175ac7fa56bb2 (DEFAULT_TABLE table=test-table [id=4c95b262ec1a4dbdbf740517878966b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:16.678136 18815 tablet_service.cc:1467] Processing CreateTablet for tablet 038f402314264aebacc175ac7fa56bb2 (DEFAULT_TABLE table=test-table [id=4c95b262ec1a4dbdbf740517878966b7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:16.680202 18624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038f402314264aebacc175ac7fa56bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:16.681172 18971 ts_tablet_manager.cc:1918] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:21:16.681489 18971 log.cc:1196] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/04198ebbf1b745508a195318e541c43a
I20260806 09:21:16.682224 18971 ts_tablet_manager.cc:1939] T 04198ebbf1b745508a195318e541c43a P 7d3a8b99c9c743dea16530c9ee41ae68: Deleting consensus metadata
I20260806 09:21:16.683189 18815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 038f402314264aebacc175ac7fa56bb2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:16.683804 18969 tablet_replica.cc:330] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11: stopping tablet replica
I20260806 09:21:16.684468 18969 raft_consensus.cc:2229] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:21:16.684724 18214 catalog_manager.cc:4841] TS 7d3a8b99c9c743dea16530c9ee41ae68 (127.16.86.130:33429): tablet 04198ebbf1b745508a195318e541c43a (table test-table [id=1e1cf2019c8b40aeb43b569c1b347dba]) successfully deleted
I20260806 09:21:16.685500 18969 raft_consensus.cc:2258] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:16.689039 18969 ts_tablet_manager.cc:1905] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:16.692648 18967 ts_tablet_manager.cc:1918] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:21:16.692896 18967 log.cc:1196] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/04198ebbf1b745508a195318e541c43a
I20260806 09:21:16.693574 18967 ts_tablet_manager.cc:1939] T 04198ebbf1b745508a195318e541c43a P 5b42eb25377842c2bec71feb76b51de3: Deleting consensus metadata
I20260806 09:21:16.696158 18214 catalog_manager.cc:4841] TS 5b42eb25377842c2bec71feb76b51de3 (127.16.86.131:36907): tablet 04198ebbf1b745508a195318e541c43a (table test-table [id=1e1cf2019c8b40aeb43b569c1b347dba]) successfully deleted
I20260806 09:21:16.696748 18976 tablet_bootstrap.cc:492] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3: Bootstrap starting.
I20260806 09:21:16.697254 18973 tablet_bootstrap.cc:492] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68: Bootstrap starting.
I20260806 09:21:16.698150 18974 tablet_bootstrap.cc:492] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11: Bootstrap starting.
I20260806 09:21:16.698896 18976 tablet_bootstrap.cc:654] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:16.699407 18973 tablet_bootstrap.cc:654] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:16.703955 18973 tablet_bootstrap.cc:492] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68: No bootstrap required, opened a new log
I20260806 09:21:16.704130 18973 ts_tablet_manager.cc:1397] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68: Time spent bootstrapping tablet: real 0.007s user 0.000s sys 0.005s
I20260806 09:21:16.704897 18974 tablet_bootstrap.cc:654] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:16.705035 18973 raft_consensus.cc:348] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.705376 18973 raft_consensus.cc:374] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:16.705487 18973 raft_consensus.cc:729] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3a8b99c9c743dea16530c9ee41ae68, State: Initialized, Role: FOLLOWER
I20260806 09:21:16.705792 18973 consensus_queue.cc:260] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.709637 18973 ts_tablet_manager.cc:1428] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260806 09:21:16.713358 18969 ts_tablet_manager.cc:1918] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:21:16.713637 18969 log.cc:1196] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/04198ebbf1b745508a195318e541c43a
I20260806 09:21:16.713912 18976 tablet_bootstrap.cc:492] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3: No bootstrap required, opened a new log
I20260806 09:21:16.714098 18976 ts_tablet_manager.cc:1397] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3: Time spent bootstrapping tablet: real 0.017s user 0.006s sys 0.000s
I20260806 09:21:16.714389 18969 ts_tablet_manager.cc:1939] T 04198ebbf1b745508a195318e541c43a P e0542bd8775b4312a6ac4205a3123e11: Deleting consensus metadata
I20260806 09:21:16.714965 18976 raft_consensus.cc:348] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.715281 18976 raft_consensus.cc:374] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:16.715392 18976 raft_consensus.cc:729] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b42eb25377842c2bec71feb76b51de3, State: Initialized, Role: FOLLOWER
I20260806 09:21:16.715682 18976 consensus_queue.cc:260] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.716349 18976 ts_tablet_manager.cc:1428] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 09:21:16.716818 18216 catalog_manager.cc:4841] TS e0542bd8775b4312a6ac4205a3123e11 (127.16.86.129:41115): tablet 04198ebbf1b745508a195318e541c43a (table test-table [id=1e1cf2019c8b40aeb43b569c1b347dba]) successfully deleted
I20260806 09:21:16.718750 18974 tablet_bootstrap.cc:492] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11: No bootstrap required, opened a new log
I20260806 09:21:16.719908 18974 ts_tablet_manager.cc:1397] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11: Time spent bootstrapping tablet: real 0.022s user 0.003s sys 0.011s
I20260806 09:21:16.721143 18974 raft_consensus.cc:348] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.721488 18974 raft_consensus.cc:374] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:16.721611 18974 raft_consensus.cc:729] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0542bd8775b4312a6ac4205a3123e11, State: Initialized, Role: FOLLOWER
I20260806 09:21:16.721907 18974 consensus_queue.cc:260] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.722718 18974 ts_tablet_manager.cc:1428] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260806 09:21:16.742707 18940 raft_consensus.cc:482] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:21:16.743104 18940 raft_consensus.cc:504] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.745198 18940 leader_election.cc:288] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0542bd8775b4312a6ac4205a3123e11 (127.16.86.129:41115), 7d3a8b99c9c743dea16530c9ee41ae68 (127.16.86.130:33429)
I20260806 09:21:16.753998 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.745632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:55765 (local address 127.16.86.130:33429)
0806 09:21:16.745860 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:16.745866 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:16.747211 (+ 1345us) server_negotiation.cc:407] Connection header received
0806 09:21:16.747440 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:16.747445 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:16.747525 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:16.747637 (+ 112us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:16.748558 (+ 921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.749670 (+ 1112us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.751680 (+ 2010us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.752764 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.752940 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.753019 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:16.753513 (+ 494us) server_negotiation.cc:299] Negotiation successful
0806 09:21:16.753670 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:21:16.754144 18983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.745540 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:33429 (local address 127.16.86.131:55765)
0806 09:21:16.747068 (+ 1528us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:16.747089 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:16.747264 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:16.747689 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:16.747694 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:16.747747 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:16.748367 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.748382 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.749817 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.749823 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.751508 (+ 1685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.751521 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.753066 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.753073 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.753292 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.753338 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 09:21:16.753451 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 09:21:16.753630 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1372,"mutex_wait_us":7,"thread_start_us":1142,"threads_started":1}
I20260806 09:21:16.754940 18652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038f402314264aebacc175ac7fa56bb2" candidate_uuid: "5b42eb25377842c2bec71feb76b51de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" is_pre_election: true
I20260806 09:21:16.755322 18982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.745509 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:41115 (local address 127.16.86.131:41589)
0806 09:21:16.746409 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:16.746437 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:16.746759 (+ 322us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:16.747511 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:16.747515 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:16.747564 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:16.748038 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.748054 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.749568 (+ 1514us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.749574 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.752014 (+ 2440us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.752032 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.753873 (+ 1841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.753880 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.754470 (+ 590us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.754512 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 09:21:16.754624 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 09:21:16.754792 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":720,"mutex_wait_us":28,"spinlock_wait_cycles":8960,"thread_start_us":776,"threads_started":1}
I20260806 09:21:16.755273 18652 raft_consensus.cc:2454] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b42eb25377842c2bec71feb76b51de3 in term 0.
I20260806 09:21:16.755337 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.746034 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:41589 (local address 127.16.86.129:41115)
0806 09:21:16.746284 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:16.746291 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:16.746570 (+ 279us) server_negotiation.cc:407] Connection header received
0806 09:21:16.746965 (+ 395us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:16.746972 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:16.747153 (+ 181us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:16.747252 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:16.748214 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.749400 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.752205 (+ 2805us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.753384 (+ 1179us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.753613 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.753689 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:16.754687 (+ 998us) server_negotiation.cc:299] Negotiation successful
0806 09:21:16.754896 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 09:21:16.756026 18743 leader_election.cc:302] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b42eb25377842c2bec71feb76b51de3, 7d3a8b99c9c743dea16530c9ee41ae68; no voters:
I20260806 09:21:16.756534 18940 raft_consensus.cc:2781] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:21:16.756686 18940 raft_consensus.cc:482] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:21:16.756798 18940 raft_consensus.cc:3037] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:16.756562 18464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038f402314264aebacc175ac7fa56bb2" candidate_uuid: "5b42eb25377842c2bec71feb76b51de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0542bd8775b4312a6ac4205a3123e11" is_pre_election: true
I20260806 09:21:16.757100 18464 raft_consensus.cc:2454] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5b42eb25377842c2bec71feb76b51de3 in term 0.
I20260806 09:21:16.760582 18940 raft_consensus.cc:504] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.761507 18940 leader_election.cc:288] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [CANDIDATE]: Term 1 election: Requested vote from peers e0542bd8775b4312a6ac4205a3123e11 (127.16.86.129:41115), 7d3a8b99c9c743dea16530c9ee41ae68 (127.16.86.130:33429)
I20260806 09:21:16.762104 18464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038f402314264aebacc175ac7fa56bb2" candidate_uuid: "5b42eb25377842c2bec71feb76b51de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0542bd8775b4312a6ac4205a3123e11"
I20260806 09:21:16.762173 18652 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "038f402314264aebacc175ac7fa56bb2" candidate_uuid: "5b42eb25377842c2bec71feb76b51de3" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7d3a8b99c9c743dea16530c9ee41ae68"
I20260806 09:21:16.762373 18464 raft_consensus.cc:3037] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:16.762452 18652 raft_consensus.cc:3037] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:16.765524 18652 raft_consensus.cc:2454] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b42eb25377842c2bec71feb76b51de3 in term 1.
I20260806 09:21:16.766022 18743 leader_election.cc:302] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b42eb25377842c2bec71feb76b51de3, 7d3a8b99c9c743dea16530c9ee41ae68; no voters:
I20260806 09:21:16.766371 18464 raft_consensus.cc:2454] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5b42eb25377842c2bec71feb76b51de3 in term 1.
I20260806 09:21:16.766772 18940 raft_consensus.cc:2781] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:16.767436 18940 raft_consensus.cc:686] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 1 LEADER]: Becoming Leader. State: Replica: 5b42eb25377842c2bec71feb76b51de3, State: Running, Role: LEADER
I20260806 09:21:16.767970 18940 consensus_queue.cc:237] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.772754 18229 catalog_manager.cc:5495] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 reported cstate change: term changed from 0 to 1, leader changed from <none> to 5b42eb25377842c2bec71feb76b51de3 (127.16.86.131). New cstate: current_term: 1 leader_uuid: "5b42eb25377842c2bec71feb76b51de3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } health_report { overall_health: HEALTHY } } }
Aug 06 09:21:16 dist-test-slave-h3ft krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008073, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:16.821393 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.801719 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41457 (local address 127.0.0.1:45610)
0806 09:21:16.802658 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:16.802685 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:16.802927 (+ 242us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:16.803440 (+ 513us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:16.803446 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:16.804598 (+ 1152us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:16.804976 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.804991 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.806585 (+ 1594us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.806592 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.808273 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:16.808294 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.809376 (+ 1082us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.809383 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:16.809586 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.810544 (+ 958us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:16.810572 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:16.814047 (+ 3475us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:16.818050 (+ 4003us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:16.818070 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:16.818082 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:16.818586 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:16.819027 (+ 441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:16.819035 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:16.819039 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:16.819225 (+ 186us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:16.819838 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:16.819849 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:16.820065 (+ 216us) client_negotiation.cc:769] Sending connection context
0806 09:21:16.820254 (+ 189us) client_negotiation.cc:240] Negotiation successful
0806 09:21:16.820764 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"mutex_wait_us":71,"thread_start_us":439,"threads_started":1}
I20260806 09:21:16.821410 18995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.801874 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45610 (local address 127.16.86.190:41457)
0806 09:21:16.802978 (+ 1104us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:16.802982 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:16.803104 (+ 122us) server_negotiation.cc:407] Connection header received
0806 09:21:16.803168 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:16.803173 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:16.803238 (+ 65us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:16.803331 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:16.805227 (+ 1896us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.806304 (+ 1077us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.808512 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:16.809139 (+ 627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:16.809282 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:16.814344 (+ 5062us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:16.814351 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:16.814355 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:16.814394 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:16.817882 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:16.818725 (+ 843us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:16.818737 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:16.818739 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:16.818883 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:16.819354 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:16.819365 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:16.819367 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:16.819687 (+ 320us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:16.819903 (+ 216us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:16.820358 (+ 455us) server_negotiation.cc:299] Negotiation successful
0806 09:21:16.820893 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":240,"server-negotiator.queue_time_us":966,"thread_start_us":800,"threads_started":1}
I20260806 09:21:16.828547 18229 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:45610:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 09:21:16.830118 18229 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:21:16.838985 18624 tablet_service.cc:1467] Processing CreateTablet for tablet e955b199e0de4ddd937182fa879dd019 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bcd412e8c0b4ecf8b7e259398ada52e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:21:16.839043 18430 tablet_service.cc:1467] Processing CreateTablet for tablet e955b199e0de4ddd937182fa879dd019 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bcd412e8c0b4ecf8b7e259398ada52e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:21:16.839357 18815 tablet_service.cc:1467] Processing CreateTablet for tablet e955b199e0de4ddd937182fa879dd019 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=0bcd412e8c0b4ecf8b7e259398ada52e]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:21:16.839736 18430 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e955b199e0de4ddd937182fa879dd019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:16.839757 18624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e955b199e0de4ddd937182fa879dd019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:16.840015 18815 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet e955b199e0de4ddd937182fa879dd019. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:16.848560 18976 tablet_bootstrap.cc:492] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3: Bootstrap starting.
I20260806 09:21:16.850446 18976 tablet_bootstrap.cc:654] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:16.856187 18974 tablet_bootstrap.cc:492] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11: Bootstrap starting.
I20260806 09:21:16.857523 18973 tablet_bootstrap.cc:492] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: Bootstrap starting.
I20260806 09:21:16.858383 18974 tablet_bootstrap.cc:654] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:16.858451 18976 tablet_bootstrap.cc:492] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3: No bootstrap required, opened a new log
I20260806 09:21:16.858632 18976 ts_tablet_manager.cc:1397] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3: Time spent bootstrapping tablet: real 0.010s user 0.002s sys 0.006s
I20260806 09:21:16.859676 18973 tablet_bootstrap.cc:654] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:16.859550 18976 raft_consensus.cc:348] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.859856 18976 raft_consensus.cc:374] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:16.859958 18976 raft_consensus.cc:729] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5b42eb25377842c2bec71feb76b51de3, State: Initialized, Role: FOLLOWER
I20260806 09:21:16.860247 18976 consensus_queue.cc:260] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.860826 18940 tablet_replica.cc:439] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } } }
I20260806 09:21:16.861289 18940 tablet_replica.cc:442] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:16.864792 18976 ts_tablet_manager.cc:1428] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260806 09:21:16.867939 18973 tablet_bootstrap.cc:492] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: No bootstrap required, opened a new log
I20260806 09:21:16.868175 18973 ts_tablet_manager.cc:1397] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: Time spent bootstrapping tablet: real 0.011s user 0.004s sys 0.005s
I20260806 09:21:16.868310 18974 tablet_bootstrap.cc:492] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11: No bootstrap required, opened a new log
I20260806 09:21:16.868518 18974 ts_tablet_manager.cc:1397] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11: Time spent bootstrapping tablet: real 0.012s user 0.000s sys 0.006s
I20260806 09:21:16.869104 18973 raft_consensus.cc:348] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.869411 18973 raft_consensus.cc:374] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:16.869516 18973 raft_consensus.cc:729] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7d3a8b99c9c743dea16530c9ee41ae68, State: Initialized, Role: FOLLOWER
I20260806 09:21:16.869480 18974 raft_consensus.cc:348] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.869964 18974 raft_consensus.cc:374] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:16.870097 18974 raft_consensus.cc:729] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e0542bd8775b4312a6ac4205a3123e11, State: Initialized, Role: FOLLOWER
I20260806 09:21:16.870393 18974 consensus_queue.cc:260] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.870534 18973 consensus_queue.cc:260] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.871258 18973 ts_tablet_manager.cc:1428] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260806 09:21:16.870934 18964 tablet_replica.cc:439] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } } }
I20260806 09:21:16.871408 18964 tablet_replica.cc:442] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:16.872285 18939 tablet_replica.cc:439] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } } }
I20260806 09:21:16.872694 18939 tablet_replica.cc:442] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:16.873180 18974 ts_tablet_manager.cc:1428] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11: Time spent starting tablet: real 0.005s user 0.000s sys 0.003s
I20260806 09:21:16.986891 18939 raft_consensus.cc:482] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:21:16.987232 18939 raft_consensus.cc:504] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:16.990618 18939 leader_election.cc:288] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e0542bd8775b4312a6ac4205a3123e11 (127.16.86.129:41115), 5b42eb25377842c2bec71feb76b51de3 (127.16.86.131:36907)
I20260806 09:21:17.010543 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.991498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:41983 (local address 127.16.86.131:36907)
0806 09:21:16.991707 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:16.991716 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:17.000287 (+ 8571us) server_negotiation.cc:407] Connection header received
0806 09:21:17.000778 (+ 491us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:17.000785 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:17.000871 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:17.000972 (+ 101us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:17.002064 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.003393 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.007490 (+ 4097us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.008907 (+ 1417us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.009045 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.009140 (+ 95us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:17.010034 (+ 894us) server_negotiation.cc:299] Negotiation successful
0806 09:21:17.010235 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 09:21:17.011164 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.990648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:54447 (local address 127.16.86.129:41115)
0806 09:21:16.991251 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:16.991258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:16.999952 (+ 8694us) server_negotiation.cc:407] Connection header received
0806 09:21:17.000281 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:17.000295 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:17.000520 (+ 225us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:17.000696 (+ 176us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:17.002033 (+ 1337us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.003471 (+ 1438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.007452 (+ 3981us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.009027 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.009222 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.009328 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:17.010144 (+ 816us) server_negotiation.cc:299] Negotiation successful
0806 09:21:17.010347 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399}
I20260806 09:21:17.011555 18999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.990157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36907 (local address 127.16.86.130:41983)
0806 09:21:17.000178 (+ 10021us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:17.000196 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:17.000576 (+ 380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:17.001227 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:17.001236 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:17.001303 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:17.001863 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:17.001876 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.003579 (+ 1703us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.003584 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:17.007248 (+ 3664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:17.007270 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.009565 (+ 2295us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.009580 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:17.009804 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.009847 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 09:21:17.010893 (+ 1046us) client_negotiation.cc:240] Negotiation successful
0806 09:21:17.011097 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9883,"mutex_wait_us":278,"spinlock_wait_cycles":8832,"thread_start_us":9146,"threads_started":1}
I20260806 09:21:17.012398 18840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e955b199e0de4ddd937182fa879dd019" candidate_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b42eb25377842c2bec71feb76b51de3" is_pre_election: true
I20260806 09:21:17.012657 19001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:16.990064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:41115 (local address 127.16.86.130:54447)
0806 09:21:16.999740 (+ 9676us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:16.999774 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:17.000055 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:17.000978 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:17.000984 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:17.001027 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:17.001720 (+ 693us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:17.001747 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.003733 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.003741 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:17.007157 (+ 3416us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:17.007179 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.009631 (+ 2452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.009639 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:17.009879 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.009926 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 09:21:17.011922 (+ 1996us) client_negotiation.cc:240] Negotiation successful
0806 09:21:17.012122 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9456,"mutex_wait_us":96,"spinlock_wait_cycles":40320,"thread_start_us":9966,"threads_started":1}
I20260806 09:21:17.012717 18840 raft_consensus.cc:2454] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d3a8b99c9c743dea16530c9ee41ae68 in term 0.
I20260806 09:21:17.013361 18553 leader_election.cc:302] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b42eb25377842c2bec71feb76b51de3, 7d3a8b99c9c743dea16530c9ee41ae68; no voters:
I20260806 09:21:17.013849 18939 raft_consensus.cc:2781] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:21:17.013973 18939 raft_consensus.cc:482] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:21:17.014055 18939 raft_consensus.cc:3037] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:17.013979 18464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e955b199e0de4ddd937182fa879dd019" candidate_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0542bd8775b4312a6ac4205a3123e11" is_pre_election: true
I20260806 09:21:17.014349 18464 raft_consensus.cc:2454] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7d3a8b99c9c743dea16530c9ee41ae68 in term 0.
I20260806 09:21:17.018421 18939 raft_consensus.cc:504] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:17.019747 18840 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e955b199e0de4ddd937182fa879dd019" candidate_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5b42eb25377842c2bec71feb76b51de3"
I20260806 09:21:17.019994 18840 raft_consensus.cc:3037] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:17.019932 18464 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "e955b199e0de4ddd937182fa879dd019" candidate_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e0542bd8775b4312a6ac4205a3123e11"
I20260806 09:21:17.020288 18464 raft_consensus.cc:3037] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:17.023213 18840 raft_consensus.cc:2454] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d3a8b99c9c743dea16530c9ee41ae68 in term 1.
I20260806 09:21:17.023870 18553 leader_election.cc:302] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5b42eb25377842c2bec71feb76b51de3, 7d3a8b99c9c743dea16530c9ee41ae68; no voters:
I20260806 09:21:17.024020 18939 leader_election.cc:288] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [CANDIDATE]: Term 1 election: Requested vote from peers e0542bd8775b4312a6ac4205a3123e11 (127.16.86.129:41115), 5b42eb25377842c2bec71feb76b51de3 (127.16.86.131:36907)
I20260806 09:21:17.024266 18939 raft_consensus.cc:2781] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:17.024461 18939 raft_consensus.cc:686] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 1 LEADER]: Becoming Leader. State: Replica: 7d3a8b99c9c743dea16530c9ee41ae68, State: Running, Role: LEADER
I20260806 09:21:17.024746 18464 raft_consensus.cc:2454] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7d3a8b99c9c743dea16530c9ee41ae68 in term 1.
I20260806 09:21:17.025024 18939 consensus_queue.cc:237] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } }
I20260806 09:21:17.028635 19002 tablet_replica.cc:439] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: TxnStatusTablet state changed. Reason: New leader 7d3a8b99c9c743dea16530c9ee41ae68. Latest consensus state: current_term: 1 leader_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } } }
I20260806 09:21:17.029026 19002 tablet_replica.cc:442] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:21:17.029840 18229 catalog_manager.cc:5495] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 reported cstate change: term changed from 0 to 1, leader changed from <none> to 7d3a8b99c9c743dea16530c9ee41ae68 (127.16.86.130). New cstate: current_term: 1 leader_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } health_report { overall_health: UNKNOWN } } }
I20260806 09:21:17.030612 19004 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 09:21:17.030915 19004 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 09:21:17.080020 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:17.073113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47672 (local address 127.16.86.130:33429)
0806 09:21:17.073348 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:17.073358 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:17.073781 (+ 423us) server_negotiation.cc:407] Connection header received
0806 09:21:17.074238 (+ 457us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:17.074243 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:17.074349 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:17.074767 (+ 418us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:17.075265 (+ 498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.076117 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.077457 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.077917 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.078059 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.078589 (+ 530us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:17.079256 (+ 667us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:17.079422 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:17.079535 (+ 113us) server_negotiation.cc:299] Negotiation successful
0806 09:21:17.079636 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:21:17.079996 18993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:17.073141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:33429 (local address 127.0.0.1:47672)
0806 09:21:17.073582 (+ 441us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:17.073618 (+ 36us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:17.073919 (+ 301us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:17.074806 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:17.074812 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:17.074829 (+ 17us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:21:17.075115 (+ 286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:17.075126 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.076280 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.076285 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:17.077304 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:17.077318 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.078084 (+ 766us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.078090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:17.078290 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.078361 (+ 71us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:21:17.079378 (+ 1017us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:21:17.079403 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:21:17.079502 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 09:21:17.079680 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 09:21:17.085531 18840 raft_consensus.cc:1261] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3 [term 1 FOLLOWER]: Refusing update from remote peer 7d3a8b99c9c743dea16530c9ee41ae68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:17.085747 18464 raft_consensus.cc:1261] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11 [term 1 FOLLOWER]: Refusing update from remote peer 7d3a8b99c9c743dea16530c9ee41ae68: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:17.086704 19002 consensus_queue.cc:1035] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:17.087895 18939 consensus_queue.cc:1035] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:17.092139 18940 tablet_replica.cc:439] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3: TxnStatusTablet state changed. Reason: New leader 7d3a8b99c9c743dea16530c9ee41ae68. Latest consensus state: current_term: 1 leader_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } } }
I20260806 09:21:17.092559 18940 tablet_replica.cc:442] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:17.095355 19002 tablet_replica.cc:439] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } } }
I20260806 09:21:17.095713 19002 tablet_replica.cc:442] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:21:17.107738 18964 tablet_replica.cc:439] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11: TxnStatusTablet state changed. Reason: New leader 7d3a8b99c9c743dea16530c9ee41ae68. Latest consensus state: current_term: 1 leader_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } } }
I20260806 09:21:17.108147 18964 tablet_replica.cc:442] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:17.108783 18940 tablet_replica.cc:439] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } } }
I20260806 09:21:17.109118 18940 tablet_replica.cc:442] T e955b199e0de4ddd937182fa879dd019 P 5b42eb25377842c2bec71feb76b51de3: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:17.109197 18939 tablet_replica.cc:439] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } } }
I20260806 09:21:17.109491 18939 tablet_replica.cc:442] T e955b199e0de4ddd937182fa879dd019 P 7d3a8b99c9c743dea16530c9ee41ae68: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:21:17.111848 18964 tablet_replica.cc:439] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 } } peers { permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 } } peers { permanent_uuid: "5b42eb25377842c2bec71feb76b51de3" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36907 } } }
I20260806 09:21:17.112090 18964 tablet_replica.cc:442] T e955b199e0de4ddd937182fa879dd019 P e0542bd8775b4312a6ac4205a3123e11: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:17.124117 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:17.117791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45980 (local address 127.16.86.131:36907)
0806 09:21:17.117971 (+ 180us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:17.117979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:17.118108 (+ 129us) server_negotiation.cc:407] Connection header received
0806 09:21:17.118451 (+ 343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:17.118457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:17.118591 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:17.118709 (+ 118us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:17.119497 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.120245 (+ 748us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.121631 (+ 1386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.122114 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.122254 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.122694 (+ 440us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:17.123337 (+ 643us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:17.123509 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:17.123701 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 09:21:17.123789 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:21:17.148612 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:17.135669 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:54215 (local address 127.16.86.130:33429)
0806 09:21:17.135937 (+ 268us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:17.135947 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:17.142993 (+ 7046us) server_negotiation.cc:407] Connection header received
0806 09:21:17.143286 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:17.143290 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:17.143367 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:17.143476 (+ 109us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:17.144103 (+ 627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.145018 (+ 915us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.146477 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.147138 (+ 661us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.147308 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.147779 (+ 471us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:17.147889 (+ 110us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:17.147993 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:17.148157 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 09:21:17.148248 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 09:21:17.148663 19021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:17.135552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:33429 (local address 127.16.86.131:54215)
0806 09:21:17.142852 (+ 7300us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:17.142871 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:17.143102 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:17.143546 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:17.143551 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:17.143563 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:21:17.143918 (+ 355us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:17.143932 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.145186 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.145191 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:17.146325 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:17.146338 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.147350 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.147358 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:17.147498 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.147562 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:21:17.147994 (+ 432us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:21:17.148009 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 09:21:17.148121 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 09:21:17.148263 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7101,"thread_start_us":6941,"threads_started":1}
I20260806 09:21:17.162616 18464 raft_consensus.cc:1261] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11 [term 1 FOLLOWER]: Refusing update from remote peer 5b42eb25377842c2bec71feb76b51de3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:17.162881 18652 raft_consensus.cc:1261] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 1 FOLLOWER]: Refusing update from remote peer 5b42eb25377842c2bec71feb76b51de3: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:17.163715 18940 consensus_queue.cc:1035] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e0542bd8775b4312a6ac4205a3123e11" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:17.164131 18984 consensus_queue.cc:1035] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7d3a8b99c9c743dea16530c9ee41ae68" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33429 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:17.185010 18952 mvcc.cc:205] Tried to move back new op lower bound from 7315489084049874944 to 7315489082447003648. Current Snapshot: MvccSnapshot[applied={T|T < 7315489084049874944}]
I20260806 09:21:17.252269 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:17.242964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:33891 (local address 127.16.86.131:36907)
0806 09:21:17.243235 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:17.243242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:17.244060 (+ 818us) server_negotiation.cc:407] Connection header received
0806 09:21:17.244435 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:17.244441 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:17.244543 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:17.244611 (+ 68us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:17.245651 (+ 1040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.246814 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.249446 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.250269 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.250390 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.251302 (+ 912us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:17.251463 (+ 161us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:17.251580 (+ 117us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:17.251793 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 09:21:17.251900 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260806 09:21:17.252700 19032 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:17.242720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36907 (local address 127.16.86.130:33891)
0806 09:21:17.243901 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:17.243927 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:17.244226 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:17.244838 (+ 612us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:17.244843 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:17.244857 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:21:17.245388 (+ 531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:17.245402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.247067 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.247074 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:17.249225 (+ 2151us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:17.249243 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.250663 (+ 1420us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.250670 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:17.250925 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.251055 (+ 130us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:21:17.251616 (+ 561us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:21:17.251631 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 09:21:17.251723 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 09:21:17.251928 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":914,"thread_start_us":2504,"threads_started":1}
I20260806 09:21:17.325718 18229 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:17.326098 18229 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45600:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:17.330495 18229 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 46e32d7285844709a162bde458a4f1bb: Sending DeleteTablet for 3 replicas of tablet 038f402314264aebacc175ac7fa56bb2
I20260806 09:21:17.332051 18430 tablet_service.cc:1514] Processing DeleteTablet for tablet 038f402314264aebacc175ac7fa56bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:17 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:54018
I20260806 09:21:17.332353 18624 tablet_service.cc:1514] Processing DeleteTablet for tablet 038f402314264aebacc175ac7fa56bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:17 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:47658
I20260806 09:21:17.333079 18815 tablet_service.cc:1514] Processing DeleteTablet for tablet 038f402314264aebacc175ac7fa56bb2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:17 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:45966
I20260806 09:21:17.355711 19036 tablet_replica.cc:330] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11: stopping tablet replica
I20260806 09:21:17.356001 19036 raft_consensus.cc:2229] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:17.356333 19036 raft_consensus.cc:2258] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:17.346940 19038 tablet_replica.cc:330] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3: stopping tablet replica
I20260806 09:21:17.356781 19038 raft_consensus.cc:2229] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:21:17.357431 19038 raft_consensus.cc:2258] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:17.358752 19038 ts_tablet_manager.cc:1905] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:17.360270 19036 ts_tablet_manager.cc:1905] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:17.362567 19037 tablet_replica.cc:330] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68: stopping tablet replica
I20260806 09:21:17.362807 19037 raft_consensus.cc:2229] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:17.363132 19037 raft_consensus.cc:2258] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:17.365299 19037 ts_tablet_manager.cc:1905] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:17.375594 19037 ts_tablet_manager.cc:1918] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:21:17.375784 19037 log.cc:1196] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/038f402314264aebacc175ac7fa56bb2
I20260806 09:21:17.376319 19037 ts_tablet_manager.cc:1939] T 038f402314264aebacc175ac7fa56bb2 P 7d3a8b99c9c743dea16530c9ee41ae68: Deleting consensus metadata
I20260806 09:21:17.377682 18214 catalog_manager.cc:4841] TS 7d3a8b99c9c743dea16530c9ee41ae68 (127.16.86.130:33429): tablet 038f402314264aebacc175ac7fa56bb2 (table test-table [id=4c95b262ec1a4dbdbf740517878966b7]) successfully deleted
I20260806 09:21:17.378074 19038 ts_tablet_manager.cc:1918] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:21:17.378230 19038 log.cc:1196] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/038f402314264aebacc175ac7fa56bb2
I20260806 09:21:17.378697 19038 ts_tablet_manager.cc:1939] T 038f402314264aebacc175ac7fa56bb2 P 5b42eb25377842c2bec71feb76b51de3: Deleting consensus metadata
I20260806 09:21:17.379840 18214 catalog_manager.cc:4841] TS 5b42eb25377842c2bec71feb76b51de3 (127.16.86.131:36907): tablet 038f402314264aebacc175ac7fa56bb2 (table test-table [id=4c95b262ec1a4dbdbf740517878966b7]) successfully deleted
I20260806 09:21:17.388383 19036 ts_tablet_manager.cc:1918] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:21:17.388604 19036 log.cc:1196] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/038f402314264aebacc175ac7fa56bb2
I20260806 09:21:17.389246 19036 ts_tablet_manager.cc:1939] T 038f402314264aebacc175ac7fa56bb2 P e0542bd8775b4312a6ac4205a3123e11: Deleting consensus metadata
I20260806 09:21:17.391424 18216 catalog_manager.cc:4841] TS e0542bd8775b4312a6ac4205a3123e11 (127.16.86.129:41115): tablet 038f402314264aebacc175ac7fa56bb2 (table test-table [id=4c95b262ec1a4dbdbf740517878966b7]) successfully deleted
Aug 06 09:21:17 dist-test-slave-h3ft krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:21:17.414340 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:17.364626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54034 (local address 127.16.86.129:41115)
0806 09:21:17.364881 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:17.364893 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:17.386359 (+ 21466us) server_negotiation.cc:407] Connection header received
0806 09:21:17.386813 (+ 454us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:17.386820 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:17.386925 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:17.387058 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:17.390401 (+ 3343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.392523 (+ 2122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.394793 (+ 2270us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.395663 (+ 870us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.395778 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.403319 (+ 7541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:17.403349 (+ 30us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:17.403380 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:17.403455 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:17.408839 (+ 5384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:17.410621 (+ 1782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:17.410635 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:17.410640 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:17.410872 (+ 232us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:17.411463 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:17.411470 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:17.411474 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:17.412130 (+ 656us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:17.412521 (+ 391us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:17.413267 (+ 746us) server_negotiation.cc:299] Negotiation successful
0806 09:21:17.413656 (+ 389us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 09:21:17.416925 18517 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54034
I20260806 09:21:17.456533 19043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:17.426520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45620 (local address 127.16.86.190:41457)
0806 09:21:17.439537 (+ 13017us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:17.439550 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:17.440297 (+ 747us) server_negotiation.cc:407] Connection header received
0806 09:21:17.440589 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:17.440595 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:17.440678 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:17.440828 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:17.442719 (+ 1891us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.443746 (+ 1027us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.445309 (+ 1563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.445843 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.446016 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.449224 (+ 3208us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:17.449230 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:17.449234 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:17.449309 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:17.452932 (+ 3623us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:17.453879 (+ 947us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:17.453889 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:17.453899 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:17.454065 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:17.454547 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:17.454554 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:17.454558 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:17.454925 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:17.455275 (+ 350us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:17.455680 (+ 405us) server_negotiation.cc:299] Negotiation successful
0806 09:21:17.456004 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12766,"thread_start_us":12501,"threads_started":1}
W20260806 09:21:17.457367 18229 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45620
I20260806 09:21:17.460481 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 18334
I20260806 09:21:17.479645 18925 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:17.462854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54042 (local address 127.16.86.129:41115)
0806 09:21:17.463140 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:17.463147 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:17.463172 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:21:17.463497 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:17.463504 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:17.463617 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:17.463691 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:17.465880 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.466732 (+ 852us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.468500 (+ 1768us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.469014 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.469187 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.472535 (+ 3348us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:17.472552 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:17.472557 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:17.472609 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:17.476283 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:17.477209 (+ 926us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:17.477214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:17.477217 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:17.477383 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:17.477852 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:17.477856 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:17.477858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:17.478205 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:17.478549 (+ 344us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:17.478862 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 09:21:17.479226 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 09:21:17.480361 18517 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:54042
W20260806 09:21:17.481124 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 18334: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:21:17.481369 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 18334
I20260806 09:21:17.520920 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 18532
Aug 06 09:21:17 dist-test-slave-h3ft krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:21:17.540810 18923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:17.521882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47678 (local address 127.16.86.130:33429)
0806 09:21:17.522128 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:17.522135 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:17.522218 (+ 83us) server_negotiation.cc:407] Connection header received
0806 09:21:17.522481 (+ 263us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:17.522486 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:17.522562 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:17.522691 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:17.524279 (+ 1588us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.525303 (+ 1024us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.526658 (+ 1355us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.527272 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.527430 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.532411 (+ 4981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:17.532431 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:17.532452 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:17.532530 (+ 78us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:17.536689 (+ 4159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:17.537852 (+ 1163us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:17.537862 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:17.537865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:17.538063 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:17.538538 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:17.538550 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:17.538552 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:17.539168 (+ 616us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:17.539642 (+ 474us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:17.539873 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 09:21:17.540182 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20260806 09:21:17.541765 18710 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47678
W20260806 09:21:17.542622 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 18532: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:21:17.542868 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 18532
I20260806 09:21:17.579571 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 18721
Aug 06 09:21:17 dist-test-slave-h3ft krb5kdc[18174](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008076, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:21:17.600900 18926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:17.580673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45988 (local address 127.16.86.131:36907)
0806 09:21:17.580955 (+ 282us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:17.580963 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:17.581127 (+ 164us) server_negotiation.cc:407] Connection header received
0806 09:21:17.581392 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:17.581397 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:17.581482 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:17.581631 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:17.583378 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.584391 (+ 1013us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.586016 (+ 1625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:17.586566 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:17.586728 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:17.592537 (+ 5809us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:17.592575 (+ 38us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:17.592600 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:17.592671 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:17.597096 (+ 4425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:17.598126 (+ 1030us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:17.598136 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:17.598139 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:17.598314 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:17.598774 (+ 460us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:17.598781 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:17.598784 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:17.599379 (+ 595us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:17.599787 (+ 408us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:17.600136 (+ 349us) server_negotiation.cc:299] Negotiation successful
0806 09:21:17.600446 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20260806 09:21:17.601922 18898 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:45988
W20260806 09:21:17.602942 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 18721: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:21:17.603255 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 18721
I20260806 09:21:17.645385 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 18193
I20260806 09:21:17.647133 18295 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:17.773788 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 18193
2026-08-06T09:21:17Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5965 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:21:17 dist-test-slave-h3ft krb5kdc[19050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:21:17 dist-test-slave-h3ft krb5kdc[19050](info): set up 2 sockets
Aug 06 09:21:17 dist-test-slave-h3ft krb5kdc[19050](info): commencing operation
krb5kdc: starting...
W20260806 09:21:19.107280 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.271s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:21:19 dist-test-slave-h3ft krb5kdc[19050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:21:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:21:19.376080 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:37049
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:37049
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:19.741976 19069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:19.742376 19069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:19.742558 19069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:19.750016 19069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:19.750207 19069 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 09:21:19.750309 19069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:19.750430 19069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:19.750530 19069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:19.750623 19069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:21:19.760975 19069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:37049
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:37049
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:19.762434 19069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:19.763908 19069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:19.780184 19076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:19.780228 19077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:19.781728 19080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:19.782353 19069 server_base.cc:1034] running on GCE node
I20260806 09:21:19.783561 19069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:19.785355 19069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:19.786648 19069 hybrid_clock.cc:648] HybridClock initialized: now 1786008079786569 us; error 46 us; skew 500 ppm
I20260806 09:21:19.787104 19069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:19 dist-test-slave-h3ft krb5kdc[19050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008079, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:19.796871 19069 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:21:19.798816 19069 webserver.cc:460] Webserver started at http://127.16.86.190:33767/ using document root <none> and password file <none>
I20260806 09:21:19.799732 19069 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:19.799885 19069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:19.800243 19069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:19.803632 19069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "00f28b1f810f4b33b8ad559ab85206a1"
format_stamp: "Formatted at 2026-08-06 09:21:19 on dist-test-slave-h3ft"
server_key: "d0778aadb92d9e9bcda07fed3afd417e"
server_key_iv: "b559f4654bcc24e48c59e864d2f591c4"
server_key_version: "encryptionkey@0"
I20260806 09:21:19.804535 19069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "00f28b1f810f4b33b8ad559ab85206a1"
format_stamp: "Formatted at 2026-08-06 09:21:19 on dist-test-slave-h3ft"
server_key: "d0778aadb92d9e9bcda07fed3afd417e"
server_key_iv: "b559f4654bcc24e48c59e864d2f591c4"
server_key_version: "encryptionkey@0"
I20260806 09:21:19.810745 19069 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260806 09:21:19.815110 19091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:19.816826 19069 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:21:19.817052 19069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "00f28b1f810f4b33b8ad559ab85206a1"
format_stamp: "Formatted at 2026-08-06 09:21:19 on dist-test-slave-h3ft"
server_key: "d0778aadb92d9e9bcda07fed3afd417e"
server_key_iv: "b559f4654bcc24e48c59e864d2f591c4"
server_key_version: "encryptionkey@0"
I20260806 09:21:19.817318 19069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:19.835171 19069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:19.840370 19069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:19.840736 19069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:19.868950 19069 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:37049
I20260806 09:21:19.869119 19176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:37049 every 8 connection(s)
I20260806 09:21:19.870625 19069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:19.875910 19177 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:19.876389 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 19069
I20260806 09:21:19.876652 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:19.877424 16730 external_mini_cluster.cc:1428] Setting key fa5da0879307b4b1e78a55c710d76b54
I20260806 09:21:19.895058 19177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1: Bootstrap starting.
Aug 06 09:21:19 dist-test-slave-h3ft krb5kdc[19050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008079, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:19.899466 19177 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:19.901226 19177 log.cc:824] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:19.905849 19177 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1: No bootstrap required, opened a new log
I20260806 09:21:19.909359 19181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:19.880094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42030 (local address 127.16.86.190:37049)
0806 09:21:19.881360 (+ 1266us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:19.881381 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:19.881463 (+ 82us) server_negotiation.cc:407] Connection header received
0806 09:21:19.885592 (+ 4129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:19.885642 (+ 50us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:19.886415 (+ 773us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:19.887063 (+ 648us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:19.888901 (+ 1838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:19.891192 (+ 2291us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:19.892702 (+ 1510us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:19.893873 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:19.894084 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:19.898884 (+ 4800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:19.898899 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:19.898934 (+ 35us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:19.898994 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:19.903712 (+ 4718us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:19.904600 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:19.904611 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:19.904622 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:19.904898 (+ 276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:19.905396 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:19.905401 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:19.905404 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:19.906291 (+ 887us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:19.906669 (+ 378us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:19.907126 (+ 457us) server_negotiation.cc:299] Negotiation successful
0806 09:21:19.907683 (+ 557us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":907,"thread_start_us":545,"threads_started":1}
I20260806 09:21:19.916924 19177 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f28b1f810f4b33b8ad559ab85206a1" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37049 } }
I20260806 09:21:19.917413 19177 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:19.917507 19177 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00f28b1f810f4b33b8ad559ab85206a1, State: Initialized, Role: FOLLOWER
I20260806 09:21:19.918169 19177 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f28b1f810f4b33b8ad559ab85206a1" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37049 } }
I20260806 09:21:19.918422 19177 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:19.918538 19177 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:19.918690 19177 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:19.921631 19177 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f28b1f810f4b33b8ad559ab85206a1" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37049 } }
I20260806 09:21:19.922353 19177 leader_election.cc:302] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00f28b1f810f4b33b8ad559ab85206a1; no voters:
I20260806 09:21:19.923200 19177 leader_election.cc:288] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:19.923470 19185 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:19.924489 19185 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [term 1 LEADER]: Becoming Leader. State: Replica: 00f28b1f810f4b33b8ad559ab85206a1, State: Running, Role: LEADER
I20260806 09:21:19.925182 19185 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f28b1f810f4b33b8ad559ab85206a1" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37049 } }
I20260806 09:21:19.926120 19177 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:19.928856 19186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00f28b1f810f4b33b8ad559ab85206a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f28b1f810f4b33b8ad559ab85206a1" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37049 } } }
I20260806 09:21:19.929410 19186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:19.929019 19187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00f28b1f810f4b33b8ad559ab85206a1. Latest consensus state: current_term: 1 leader_uuid: "00f28b1f810f4b33b8ad559ab85206a1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00f28b1f810f4b33b8ad559ab85206a1" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37049 } } }
I20260806 09:21:19.929787 19187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:19.932755 19194 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:19.941159 19194 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:19.954232 19194 catalog_manager.cc:1347] Generated new cluster ID: 6637780c3e2e4be08895dfd4067d2fd7
I20260806 09:21:19.954432 19194 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:19.987922 19194 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:19.989678 19194 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:20.000689 19194 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1: Generated new TSK 0
I20260806 09:21:20.001881 19194 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:21:20.075697 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:37049
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:20.493368 19214 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:20.493783 19214 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:20.494038 19214 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:20.502835 19214 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:20.503053 19214 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:20.503239 19214 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 09:21:20.503315 19214 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:21:20.503444 19214 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:21:20.515302 19214 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.86.190:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:20.517128 19214 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:20.518873 19214 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:20.536005 19223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:20.536062 19224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:20.536579 19226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:20.537460 19214 server_base.cc:1034] running on GCE node
I20260806 09:21:20.538293 19214 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:20.539666 19214 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:20.540967 19214 hybrid_clock.cc:648] HybridClock initialized: now 1786008080540908 us; error 44 us; skew 500 ppm
I20260806 09:21:20.541414 19214 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:20 dist-test-slave-h3ft krb5kdc[19050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008080, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:20.545603 19214 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:21:20.547669 19214 webserver.cc:460] Webserver started at http://127.16.86.129:37293/ using document root <none> and password file <none>
I20260806 09:21:20.548565 19214 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:20.548728 19214 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:20.549121 19214 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:20.552850 19214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "dab79d1f499840e59dd96ecd7cd9108b"
format_stamp: "Formatted at 2026-08-06 09:21:20 on dist-test-slave-h3ft"
server_key: "1776f15414bd02ad69ed1b5b14b1c275"
server_key_iv: "5a84403c2625c0d179385d0d5c01355a"
server_key_version: "encryptionkey@0"
I20260806 09:21:20.553728 19214 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "dab79d1f499840e59dd96ecd7cd9108b"
format_stamp: "Formatted at 2026-08-06 09:21:20 on dist-test-slave-h3ft"
server_key: "1776f15414bd02ad69ed1b5b14b1c275"
server_key_iv: "5a84403c2625c0d179385d0d5c01355a"
server_key_version: "encryptionkey@0"
I20260806 09:21:20.560302 19214 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 09:21:20.564640 19234 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:20.566484 19214 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:21:20.566728 19214 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "dab79d1f499840e59dd96ecd7cd9108b"
format_stamp: "Formatted at 2026-08-06 09:21:20 on dist-test-slave-h3ft"
server_key: "1776f15414bd02ad69ed1b5b14b1c275"
server_key_iv: "5a84403c2625c0d179385d0d5c01355a"
server_key_version: "encryptionkey@0"
I20260806 09:21:20.567003 19214 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:20.595077 19214 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:20.600754 19214 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:20.601193 19214 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:20.604928 19214 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:20.605087 19214 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:20.605229 19214 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:20.605352 19214 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:21:20 dist-test-slave-h3ft krb5kdc[19050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008080, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:20.631037 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:20.605561 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:47549 (local address 127.16.86.190:37049)
0806 09:21:20.606528 (+ 967us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:20.606533 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:20.607962 (+ 1429us) server_negotiation.cc:407] Connection header received
0806 09:21:20.610332 (+ 2370us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:20.610338 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:20.610461 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:20.610616 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:20.613239 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:20.614723 (+ 1484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:20.617834 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:20.618481 (+ 647us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:20.618643 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:20.623740 (+ 5097us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:20.623747 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:20.623753 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:20.623799 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:20.627763 (+ 3964us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:20.628532 (+ 769us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:20.628538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:20.628542 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:20.628675 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:20.629142 (+ 467us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:20.629147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:20.629150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:20.629440 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:20.629599 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:20.630416 (+ 817us) server_negotiation.cc:299] Negotiation successful
0806 09:21:20.630647 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":840,"thread_start_us":689,"threads_started":1}
I20260806 09:21:20.632501 19249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:20.606595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:37049 (local address 127.16.86.129:47549)
0806 09:21:20.607760 (+ 1165us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:20.607823 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:20.610007 (+ 2184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:20.610924 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:20.610933 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:20.611897 (+ 964us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:20.612931 (+ 1034us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:20.612966 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:20.614998 (+ 2032us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:20.615037 (+ 39us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:20.617605 (+ 2568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:20.617623 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:20.618819 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:20.618829 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:20.619164 (+ 335us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:20.620089 (+ 925us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:20.620120 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:20.623398 (+ 3278us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:20.627920 (+ 4522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:20.627936 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:20.627954 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:20.628393 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:20.628790 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:20.628795 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:20.628799 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:20.629009 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:20.629574 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:20.629582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:20.629990 (+ 408us) client_negotiation.cc:769] Sending connection context
0806 09:21:20.630305 (+ 315us) client_negotiation.cc:240] Negotiation successful
0806 09:21:20.630857 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":549,"threads_started":1}
I20260806 09:21:20.664988 19248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:20.642729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:58305 (local address 127.16.86.190:37049)
0806 09:21:20.643047 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:20.643057 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:20.644050 (+ 993us) server_negotiation.cc:407] Connection header received
0806 09:21:20.644475 (+ 425us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:20.644485 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:20.644601 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:20.644745 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:20.647088 (+ 2343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:20.648469 (+ 1381us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:20.651983 (+ 3514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:20.652878 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:20.653081 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:20.656199 (+ 3118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:20.656209 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:20.656217 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:20.656277 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:20.660777 (+ 4500us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:20.662144 (+ 1367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:20.662153 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:20.662159 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:20.662374 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:20.662965 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:20.662971 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:20.662976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:20.663369 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:20.663520 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:20.664168 (+ 648us) server_negotiation.cc:299] Negotiation successful
0806 09:21:20.664500 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:21:20.665798 19405 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:33115 every 8 connection(s)
I20260806 09:21:20.665787 19343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:20.642669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:37049 (local address 127.16.86.129:58305)
0806 09:21:20.643874 (+ 1205us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:20.643900 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:20.644229 (+ 329us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:20.645054 (+ 825us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:20.645061 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:20.645967 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:20.646772 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:20.646796 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:20.648693 (+ 1897us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:20.648701 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:20.651706 (+ 3005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:20.651739 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:20.653224 (+ 1485us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:20.653235 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:20.653468 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:20.654558 (+ 1090us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:20.654592 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:20.655891 (+ 1299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:20.660985 (+ 5094us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:20.660994 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:20.661004 (+ 10us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:20.661930 (+ 926us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:20.662538 (+ 608us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:20.662546 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:20.662552 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:20.662788 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:20.663517 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:20.663522 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:20.663786 (+ 264us) client_negotiation.cc:769] Sending connection context
0806 09:21:20.663997 (+ 211us) client_negotiation.cc:240] Negotiation successful
0806 09:21:20.664568 (+ 571us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1003,"thread_start_us":799,"threads_started":1}
I20260806 09:21:20.665787 19214 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:33115
I20260806 09:21:20.667402 19214 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:21:20.668789 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 19214
I20260806 09:21:20.669057 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:21:20.669937 16730 external_mini_cluster.cc:1428] Setting key 3d5cdb7e3e97288743c731713e9be85f
I20260806 09:21:20.671612 19408 heartbeater.cc:347] Connected to a master server at 127.16.86.190:37049
I20260806 09:21:20.671900 19408 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:20.672765 19408 heartbeater.cc:511] Master 127.16.86.190:37049 requested a full tablet report, sending...
I20260806 09:21:20.675309 19114 ts_manager.cc:194] Registered new tserver with Master: dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115)
I20260806 09:21:20.677042 19114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:47549
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:21:20.728713 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:37049
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:21.111488 19412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:21.111886 19412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:21.112106 19412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:21.119709 19412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:21.119885 19412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:21.120038 19412 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 09:21:21.120133 19412 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:21:21.120278 19412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:21:21.131104 19412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.86.190:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:21.132606 19412 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:21.134125 19412 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:21.149933 19421 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:21.150136 19420 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:21.150308 19423 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:21.151862 19412 server_base.cc:1034] running on GCE node
I20260806 09:21:21.152827 19412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:21.154188 19412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:21.155441 19412 hybrid_clock.cc:648] HybridClock initialized: now 1786008081155393 us; error 45 us; skew 500 ppm
I20260806 09:21:21.155871 19412 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:21 dist-test-slave-h3ft krb5kdc[19050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008081, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:21.160023 19412 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:21:21.162034 19412 webserver.cc:460] Webserver started at http://127.16.86.130:39459/ using document root <none> and password file <none>
I20260806 09:21:21.162930 19412 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:21.163137 19412 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:21.163548 19412 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:21.167300 19412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "12c6de8415a041d68420e8cad1fac790"
format_stamp: "Formatted at 2026-08-06 09:21:21 on dist-test-slave-h3ft"
server_key: "704b9f1bca113cd64b193e0ffa40407f"
server_key_iv: "626c6911fb388a4f20811509f007eb68"
server_key_version: "encryptionkey@0"
I20260806 09:21:21.168202 19412 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "12c6de8415a041d68420e8cad1fac790"
format_stamp: "Formatted at 2026-08-06 09:21:21 on dist-test-slave-h3ft"
server_key: "704b9f1bca113cd64b193e0ffa40407f"
server_key_iv: "626c6911fb388a4f20811509f007eb68"
server_key_version: "encryptionkey@0"
I20260806 09:21:21.175405 19412 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260806 09:21:21.179698 19436 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:21.181377 19412 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:21:21.181602 19412 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "12c6de8415a041d68420e8cad1fac790"
format_stamp: "Formatted at 2026-08-06 09:21:21 on dist-test-slave-h3ft"
server_key: "704b9f1bca113cd64b193e0ffa40407f"
server_key_iv: "626c6911fb388a4f20811509f007eb68"
server_key_version: "encryptionkey@0"
I20260806 09:21:21.181857 19412 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:21.202399 19412 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:21.207942 19412 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:21.208377 19412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:21.212205 19412 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:21.212384 19412 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:21.212565 19412 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:21.213086 19412 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:21:21 dist-test-slave-h3ft krb5kdc[19050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008081, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:21.241823 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:21.212850 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:36463 (local address 127.16.86.190:37049)
0806 09:21:21.213583 (+ 733us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:21.213595 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:21.215378 (+ 1783us) server_negotiation.cc:407] Connection header received
0806 09:21:21.217716 (+ 2338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:21.217723 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:21.217819 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:21.217953 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:21.220196 (+ 2243us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.221302 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.225843 (+ 4541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.226398 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.226568 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:21.233535 (+ 6967us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:21.233545 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:21.233552 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:21.233613 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:21.237825 (+ 4212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:21.239037 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:21.239044 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:21.239048 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:21.239248 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:21.239849 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:21.239854 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:21.239858 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:21.240160 (+ 302us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:21.240341 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:21.241138 (+ 797us) server_negotiation.cc:299] Negotiation successful
0806 09:21:21.241395 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":785,"threads_started":1}
I20260806 09:21:21.242964 19452 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:21.214014 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:37049 (local address 127.16.86.130:36463)
0806 09:21:21.215159 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:21.215211 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:21.217398 (+ 2187us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:21.218193 (+ 795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:21.218202 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:21.218996 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:21.219967 (+ 971us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:21.220007 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.221571 (+ 1564us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.221583 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:21.225576 (+ 3993us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:21.225604 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.226767 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.226780 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:21.227258 (+ 478us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:21.228203 (+ 945us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:21.228245 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:21.233130 (+ 4885us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:21.238006 (+ 4876us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:21.238018 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:21.238045 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:21.238872 (+ 827us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:21.239413 (+ 541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:21.239420 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:21.239428 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:21.239700 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:21.240299 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:21.240308 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:21.240735 (+ 427us) client_negotiation.cc:769] Sending connection context
0806 09:21:21.241020 (+ 285us) client_negotiation.cc:240] Negotiation successful
0806 09:21:21.241627 (+ 607us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"thread_start_us":456,"threads_started":1}
I20260806 09:21:21.267822 19447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:21.250576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:35409 (local address 127.16.86.190:37049)
0806 09:21:21.250798 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:21.250803 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:21.252032 (+ 1229us) server_negotiation.cc:407] Connection header received
0806 09:21:21.252320 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:21.252331 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:21.252511 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:21.252705 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:21.253814 (+ 1109us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.255477 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.257948 (+ 2471us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.258682 (+ 734us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.258837 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:21.261819 (+ 2982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:21.261825 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:21.261828 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:21.261863 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:21.264925 (+ 3062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:21.265647 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:21.265655 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:21.265663 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:21.265782 (+ 119us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:21.266157 (+ 375us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:21.266164 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:21.266169 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:21.266603 (+ 434us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:21.266765 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:21.267228 (+ 463us) server_negotiation.cc:299] Negotiation successful
0806 09:21:21.267477 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:21:21.268030 19538 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:21.250631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:37049 (local address 127.16.86.130:35409)
0806 09:21:21.251824 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:21.251859 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:21.252086 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:21.252779 (+ 693us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:21.252783 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:21.253286 (+ 503us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:21.253613 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:21.253623 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.255708 (+ 2085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.255720 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:21.257647 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:21.257671 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.258918 (+ 1247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.258928 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:21.259132 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:21.260157 (+ 1025us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:21.260244 (+ 87us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:21.261582 (+ 1338us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:21.265070 (+ 3488us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:21.265077 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:21.265081 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:21.265506 (+ 425us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:21.265900 (+ 394us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:21.265905 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:21.265909 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:21.266040 (+ 131us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:21.266729 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:21.266734 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:21.266924 (+ 190us) client_negotiation.cc:769] Sending connection context
0806 09:21:21.267122 (+ 198us) client_negotiation.cc:240] Negotiation successful
0806 09:21:21.267548 (+ 426us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":951,"thread_start_us":601,"threads_started":1}
I20260806 09:21:21.283888 19412 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:38871
I20260806 09:21:21.283926 19600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:38871 every 8 connection(s)
I20260806 09:21:21.285615 19412 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:21:21.287292 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 19412
I20260806 09:21:21.287523 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:21:21.288080 16730 external_mini_cluster.cc:1428] Setting key 5a61b531e03b16fc61331425d06a6a55
I20260806 09:21:21.288935 19601 heartbeater.cc:347] Connected to a master server at 127.16.86.190:37049
I20260806 09:21:21.289235 19601 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:21.290177 19601 heartbeater.cc:511] Master 127.16.86.190:37049 requested a full tablet report, sending...
I20260806 09:21:21.291957 19114 ts_manager.cc:194] Registered new tserver with Master: 12c6de8415a041d68420e8cad1fac790 (127.16.86.130:38871)
I20260806 09:21:21.293035 19114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:36463
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:21:21.346243 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:37049
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:45205
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:21:21.680900 19408 heartbeater.cc:503] Master 127.16.86.190:37049 was elected leader, sending a full tablet report...
W20260806 09:21:21.743654 19607 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:21.744135 19607 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:21.744335 19607 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:21.752589 19607 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:21.752822 19607 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:21.753012 19607 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20260806 09:21:21.753115 19607 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:21:21.753274 19607 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:21:21.764701 19607 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:45205
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.86.190:37049
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:21.766608 19607 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:21.768463 19607 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:21.785988 19614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:21.785928 19618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:21.785934 19613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:21.786931 19607 server_base.cc:1034] running on GCE node
I20260806 09:21:21.788014 19607 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:21.789556 19607 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:21.790843 19607 hybrid_clock.cc:648] HybridClock initialized: now 1786008081790789 us; error 48 us; skew 500 ppm
I20260806 09:21:21.791332 19607 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:21 dist-test-slave-h3ft krb5kdc[19050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008081, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:21.795845 19607 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:21:21.797933 19607 webserver.cc:460] Webserver started at http://127.16.86.131:32825/ using document root <none> and password file <none>
I20260806 09:21:21.798883 19607 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:21.799072 19607 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:21.799511 19607 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:21.804142 19607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "d53f162523a64d2e941a0faa78c2b12d"
format_stamp: "Formatted at 2026-08-06 09:21:21 on dist-test-slave-h3ft"
server_key: "106e6b62dbfce67a8364b871c5a8b9f7"
server_key_iv: "14f628cbe4b2ab22a747dece023e2fca"
server_key_version: "encryptionkey@0"
I20260806 09:21:21.805310 19607 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "d53f162523a64d2e941a0faa78c2b12d"
format_stamp: "Formatted at 2026-08-06 09:21:21 on dist-test-slave-h3ft"
server_key: "106e6b62dbfce67a8364b871c5a8b9f7"
server_key_iv: "14f628cbe4b2ab22a747dece023e2fca"
server_key_version: "encryptionkey@0"
I20260806 09:21:21.812608 19607 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260806 09:21:21.817667 19627 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:21.819669 19607 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:21:21.819873 19607 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "d53f162523a64d2e941a0faa78c2b12d"
format_stamp: "Formatted at 2026-08-06 09:21:21 on dist-test-slave-h3ft"
server_key: "106e6b62dbfce67a8364b871c5a8b9f7"
server_key_iv: "14f628cbe4b2ab22a747dece023e2fca"
server_key_version: "encryptionkey@0"
I20260806 09:21:21.820109 19607 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:21.858808 19607 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:21.864470 19607 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:21.864806 19607 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:21.868546 19607 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:21.868705 19607 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:21.868891 19607 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:21.869002 19607 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:21:21 dist-test-slave-h3ft krb5kdc[19050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008081, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:21.908852 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:21.869323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:57257 (local address 127.16.86.190:37049)
0806 09:21:21.880416 (+ 11093us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:21.880428 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:21.882070 (+ 1642us) server_negotiation.cc:407] Connection header received
0806 09:21:21.884862 (+ 2792us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:21.884868 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:21.884976 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:21.885152 (+ 176us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:21.888127 (+ 2975us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.889703 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.892874 (+ 3171us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.893926 (+ 1052us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.894149 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:21.899085 (+ 4936us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:21.899096 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:21.899105 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:21.899179 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:21.903900 (+ 4721us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:21.905531 (+ 1631us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:21.905536 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:21.905539 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:21.905730 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:21.906568 (+ 838us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:21.906572 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:21.906574 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:21.906984 (+ 410us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:21.907198 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:21.908142 (+ 944us) server_negotiation.cc:299] Negotiation successful
0806 09:21:21.908414 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10876,"thread_start_us":11121,"threads_started":1}
I20260806 09:21:21.910488 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:21.870282 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:37049 (local address 127.16.86.131:57257)
0806 09:21:21.881849 (+ 11567us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:21.881918 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:21.884524 (+ 2606us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:21.885478 (+ 954us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:21.885496 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:21.886414 (+ 918us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:21.887813 (+ 1399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:21.887852 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.889956 (+ 2104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.889966 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:21.892647 (+ 2681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:21.892669 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.894240 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.894246 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:21.894447 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:21.895317 (+ 870us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:21.895346 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:21.898696 (+ 3350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:21.904137 (+ 5441us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:21.904156 (+ 19us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:21.904193 (+ 37us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:21.905305 (+ 1112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:21.905872 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:21.905886 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:21.905894 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:21.906385 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:21.907218 (+ 833us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:21.907247 (+ 29us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:21.907716 (+ 469us) client_negotiation.cc:769] Sending connection context
0806 09:21:21.908030 (+ 314us) client_negotiation.cc:240] Negotiation successful
0806 09:21:21.908695 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11203,"thread_start_us":10842,"threads_started":1}
I20260806 09:21:21.939988 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:21.917384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:36657 (local address 127.16.86.190:37049)
0806 09:21:21.917592 (+ 208us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:21.917597 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:21.919312 (+ 1715us) server_negotiation.cc:407] Connection header received
0806 09:21:21.919505 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:21.919515 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:21.919662 (+ 147us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:21.919839 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:21.920865 (+ 1026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.922768 (+ 1903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.925434 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.926524 (+ 1090us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.926760 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:21.931164 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:21.931175 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:21.931183 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:21.931245 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:21.935397 (+ 4152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:21.936967 (+ 1570us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:21.936972 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:21.936975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:21.937232 (+ 257us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:21.937830 (+ 598us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:21.937841 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:21.937843 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:21.938193 (+ 350us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:21.938326 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:21.939270 (+ 944us) server_negotiation.cc:299] Negotiation successful
0806 09:21:21.939718 (+ 448us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:21:21.940826 19711 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:21.917882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:37049 (local address 127.16.86.131:36657)
0806 09:21:21.919158 (+ 1276us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:21.919175 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:21.919343 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:21.919858 (+ 515us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:21.919862 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:21.920334 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:21.920676 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:21.920687 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.923048 (+ 2361us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.923054 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:21.925122 (+ 2068us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:21.925140 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:21.927211 (+ 2071us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:21.927223 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:21.927636 (+ 413us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:21.928770 (+ 1134us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:21.928804 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:21.930789 (+ 1985us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:21.935706 (+ 4917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:21.935713 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:21.935718 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:21.936752 (+ 1034us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:21.937393 (+ 641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:21.937404 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:21.937411 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:21.937657 (+ 246us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:21.938346 (+ 689us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:21.938356 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:21.938738 (+ 382us) client_negotiation.cc:769] Sending connection context
0806 09:21:21.938986 (+ 248us) client_negotiation.cc:240] Negotiation successful
0806 09:21:21.939784 (+ 798us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1086,"thread_start_us":749,"threads_started":1}
I20260806 09:21:21.958499 19607 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:36893
I20260806 09:21:21.958528 19798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:36893 every 8 connection(s)
I20260806 09:21:21.960341 19607 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:21:21.963665 19799 heartbeater.cc:347] Connected to a master server at 127.16.86.190:37049
I20260806 09:21:21.963986 19799 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:21.964880 19799 heartbeater.cc:511] Master 127.16.86.190:37049 requested a full tablet report, sending...
I20260806 09:21:21.965413 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 19607
I20260806 09:21:21.965643 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:21:21.966255 16730 external_mini_cluster.cc:1428] Setting key 3a444148f1d6cc50a94e925bef8293dd
I20260806 09:21:21.966746 19114 ts_manager.cc:194] Registered new tserver with Master: d53f162523a64d2e941a0faa78c2b12d (127.16.86.131:36893)
I20260806 09:21:21.968072 19114 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:57257
I20260806 09:21:21.970049 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:21:22 dist-test-slave-h3ft krb5kdc[19050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:21:22 dist-test-slave-h3ft krb5kdc[19050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008082, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:22.038672 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.021492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42038 (local address 127.16.86.190:37049)
0806 09:21:22.021746 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.021752 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.021936 (+ 184us) server_negotiation.cc:407] Connection header received
0806 09:21:22.022247 (+ 311us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.022252 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.022355 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.022489 (+ 134us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:22.024138 (+ 1649us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.025005 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.026691 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.027236 (+ 545us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.027388 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.032460 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:22.032466 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:22.032470 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:22.032519 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:22.035771 (+ 3252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:22.036670 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:22.036674 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:22.036676 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:22.036820 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:22.037216 (+ 396us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:22.037223 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:22.037228 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:22.037571 (+ 343us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:22.037863 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.038183 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.038370 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 09:21:22.043895 19114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:21:22.052639 19114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:21:22.093470 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.069784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44450 (local address 127.16.86.131:36893)
0806 09:21:22.081760 (+ 11976us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.081772 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.083100 (+ 1328us) server_negotiation.cc:407] Connection header received
0806 09:21:22.083243 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.083255 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.083639 (+ 384us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.084883 (+ 1244us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:22.085233 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.086952 (+ 1719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.090292 (+ 3340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.091890 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.092054 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.092341 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.092807 (+ 466us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.093041 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11755,"thread_start_us":11555,"threads_started":1}
I20260806 09:21:22.093485 19815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.069124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36893 (local address 127.0.0.1:44450)
0806 09:21:22.082388 (+ 13264us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.082466 (+ 78us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.082797 (+ 331us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.083959 (+ 1162us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.083970 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.084013 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:22.084612 (+ 599us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.084631 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.087168 (+ 2537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.087173 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.090054 (+ 2881us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.090073 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.092156 (+ 2083us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.092166 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.092418 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.092525 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.092716 (+ 191us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.093013 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12872,"mutex_wait_us":4,"thread_start_us":16795,"threads_started":1}
I20260806 09:21:22.095594 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.069406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43296 (local address 127.16.86.129:33115)
0806 09:21:22.086267 (+ 16861us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.086277 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.086307 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:21:22.086403 (+ 96us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.086411 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.086625 (+ 214us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.086808 (+ 183us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:22.087935 (+ 1127us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.089483 (+ 1548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.092475 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.094058 (+ 1583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.094212 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.094423 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.095117 (+ 694us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.095295 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16303,"thread_start_us":16145,"threads_started":1}
I20260806 09:21:22.096894 19816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.069137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:33115 (local address 127.0.0.1:43296)
0806 09:21:22.085182 (+ 16045us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.085203 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.085325 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.087133 (+ 1808us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.087141 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.087235 (+ 94us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:22.087736 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.087748 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.089729 (+ 1981us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.089740 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.092292 (+ 2552us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.092308 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.094594 (+ 2286us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.094614 (+ 20us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.094838 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.094877 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.096088 (+ 1211us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.096358 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15901,"mutex_wait_us":44,"thread_start_us":12693,"threads_started":1}
I20260806 09:21:22.099196 19695 tablet_service.cc:1467] Processing CreateTablet for tablet fcb32ecc99ea44cc9acdd6c97721dc5d (DEFAULT_TABLE table=test-table [id=7c7365359c9242aa8387eaf1e89539f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:22.100772 19820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.086134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:38871 (local address 127.0.0.1:60902)
0806 09:21:22.090535 (+ 4401us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.090555 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.091211 (+ 656us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.092397 (+ 1186us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.092405 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.092433 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:22.092971 (+ 538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.092983 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.095745 (+ 2762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.095753 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.097491 (+ 1738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.097504 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.099726 (+ 2222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.099733 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.099967 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.100013 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.100125 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.100307 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4227,"mutex_wait_us":54,"thread_start_us":3871,"threads_started":1}
I20260806 09:21:22.101307 19695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb32ecc99ea44cc9acdd6c97721dc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:22.101687 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.084434 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60902 (local address 127.16.86.130:38871)
0806 09:21:22.089555 (+ 5121us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.089565 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.091362 (+ 1797us) server_negotiation.cc:407] Connection header received
0806 09:21:22.091678 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.091687 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.091929 (+ 242us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.092159 (+ 230us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:22.093208 (+ 1049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.095560 (+ 2352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.097676 (+ 2116us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.099475 (+ 1799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.099646 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.099847 (+ 201us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.101115 (+ 1268us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.101369 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4915,"thread_start_us":4721,"threads_started":1}
I20260806 09:21:22.101504 19309 tablet_service.cc:1467] Processing CreateTablet for tablet fcb32ecc99ea44cc9acdd6c97721dc5d (DEFAULT_TABLE table=test-table [id=7c7365359c9242aa8387eaf1e89539f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:22.103286 19309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb32ecc99ea44cc9acdd6c97721dc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:22.105964 19503 tablet_service.cc:1467] Processing CreateTablet for tablet fcb32ecc99ea44cc9acdd6c97721dc5d (DEFAULT_TABLE table=test-table [id=7c7365359c9242aa8387eaf1e89539f3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:22.107479 19503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fcb32ecc99ea44cc9acdd6c97721dc5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:22.117043 19821 tablet_bootstrap.cc:492] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d: Bootstrap starting.
I20260806 09:21:22.120923 19821 tablet_bootstrap.cc:654] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:22.122033 19824 tablet_bootstrap.cc:492] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b: Bootstrap starting.
I20260806 09:21:22.122916 19821 log.cc:824] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:22.123629 19825 tablet_bootstrap.cc:492] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790: Bootstrap starting.
I20260806 09:21:22.127802 19825 tablet_bootstrap.cc:654] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:22.128105 19821 tablet_bootstrap.cc:492] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d: No bootstrap required, opened a new log
I20260806 09:21:22.128234 19824 tablet_bootstrap.cc:654] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:22.128458 19821 ts_tablet_manager.cc:1397] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260806 09:21:22.129150 19825 log.cc:824] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:22.129714 19824 log.cc:824] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:22.132401 19825 tablet_bootstrap.cc:492] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790: No bootstrap required, opened a new log
I20260806 09:21:22.132748 19825 ts_tablet_manager.cc:1397] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260806 09:21:22.134423 19824 tablet_bootstrap.cc:492] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b: No bootstrap required, opened a new log
I20260806 09:21:22.134761 19824 ts_tablet_manager.cc:1397] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b: Time spent bootstrapping tablet: real 0.013s user 0.007s sys 0.000s
I20260806 09:21:22.140089 19821 raft_consensus.cc:348] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } }
I20260806 09:21:22.140645 19821 raft_consensus.cc:374] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:22.140762 19821 raft_consensus.cc:729] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53f162523a64d2e941a0faa78c2b12d, State: Initialized, Role: FOLLOWER
I20260806 09:21:22.140885 19825 raft_consensus.cc:348] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } }
I20260806 09:21:22.141404 19825 raft_consensus.cc:374] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:22.141537 19825 raft_consensus.cc:729] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c6de8415a041d68420e8cad1fac790, State: Initialized, Role: FOLLOWER
I20260806 09:21:22.141534 19821 consensus_queue.cc:260] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } }
I20260806 09:21:22.142419 19825 consensus_queue.cc:260] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } }
I20260806 09:21:22.143776 19821 ts_tablet_manager.cc:1428] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d: Time spent starting tablet: real 0.015s user 0.016s sys 0.000s
I20260806 09:21:22.144380 19799 heartbeater.cc:503] Master 127.16.86.190:37049 was elected leader, sending a full tablet report...
I20260806 09:21:22.146028 19824 raft_consensus.cc:348] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } }
I20260806 09:21:22.146586 19824 raft_consensus.cc:374] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:22.146675 19824 raft_consensus.cc:729] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab79d1f499840e59dd96ecd7cd9108b, State: Initialized, Role: FOLLOWER
I20260806 09:21:22.147329 19824 consensus_queue.cc:260] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } }
I20260806 09:21:22.149471 19824 ts_tablet_manager.cc:1428] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b: Time spent starting tablet: real 0.015s user 0.011s sys 0.005s
I20260806 09:21:22.151010 19825 ts_tablet_manager.cc:1428] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790: Time spent starting tablet: real 0.018s user 0.012s sys 0.004s
I20260806 09:21:22.151432 19601 heartbeater.cc:503] Master 127.16.86.190:37049 was elected leader, sending a full tablet report...
I20260806 09:21:22.155427 19836 raft_consensus.cc:482] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:21:22.155776 19836 raft_consensus.cc:504] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } }
I20260806 09:21:22.158242 19836 leader_election.cc:288] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d53f162523a64d2e941a0faa78c2b12d (127.16.86.131:36893), 12c6de8415a041d68420e8cad1fac790 (127.16.86.130:38871)
I20260806 09:21:22.166833 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.158694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:38023 (local address 127.16.86.130:38871)
0806 09:21:22.158953 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.158961 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.160404 (+ 1443us) server_negotiation.cc:407] Connection header received
0806 09:21:22.160540 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.160546 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.160646 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.160768 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:22.161344 (+ 576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.162561 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.164805 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.165658 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.165896 (+ 238us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.165988 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.166396 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.166534 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:21:22.166956 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.158761 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:49131 (local address 127.16.86.131:36893)
0806 09:21:22.158992 (+ 231us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.159028 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.159544 (+ 516us) server_negotiation.cc:407] Connection header received
0806 09:21:22.159773 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.159778 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.159893 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.160044 (+ 151us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:22.160908 (+ 864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.162170 (+ 1262us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.164700 (+ 2530us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.165622 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.165858 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.165968 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.166439 (+ 471us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.166593 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 09:21:22.167125 19839 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.158504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:38871 (local address 127.16.86.129:38023)
0806 09:21:22.160281 (+ 1777us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.160295 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.160414 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.160803 (+ 389us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.160806 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.160827 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:22.161177 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.161186 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.162746 (+ 1560us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.162750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.164564 (+ 1814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.164578 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.165993 (+ 1415us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.166001 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.166192 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.166255 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.166367 (+ 112us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.166506 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1607,"mutex_wait_us":311,"thread_start_us":1137,"threads_started":1}
I20260806 09:21:22.167551 19837 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.158504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36893 (local address 127.16.86.129:49131)
0806 09:21:22.159375 (+ 871us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.159403 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.159605 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.160086 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.160097 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.160136 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:22.160699 (+ 563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.160719 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.162389 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.162395 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.164517 (+ 2122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.164536 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.165932 (+ 1396us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.165940 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.166132 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.166258 (+ 126us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.166407 (+ 149us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.166580 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":676,"mutex_wait_us":75,"thread_start_us":397,"threads_started":1}
I20260806 09:21:22.168406 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb32ecc99ea44cc9acdd6c97721dc5d" candidate_uuid: "dab79d1f499840e59dd96ecd7cd9108b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c6de8415a041d68420e8cad1fac790" is_pre_election: true
I20260806 09:21:22.168725 19732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb32ecc99ea44cc9acdd6c97721dc5d" candidate_uuid: "dab79d1f499840e59dd96ecd7cd9108b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53f162523a64d2e941a0faa78c2b12d" is_pre_election: true
I20260806 09:21:22.169095 19532 raft_consensus.cc:2454] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dab79d1f499840e59dd96ecd7cd9108b in term 0.
I20260806 09:21:22.169334 19732 raft_consensus.cc:2454] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dab79d1f499840e59dd96ecd7cd9108b in term 0.
I20260806 09:21:22.170029 19237 leader_election.cc:302] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c6de8415a041d68420e8cad1fac790, dab79d1f499840e59dd96ecd7cd9108b; no voters:
I20260806 09:21:22.170604 19836 raft_consensus.cc:2781] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:21:22.170797 19836 raft_consensus.cc:482] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:21:22.170940 19836 raft_consensus.cc:3037] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Advancing to term 1
W20260806 09:21:22.171588 19409 tablet.cc:2369] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:21:22.174468 19836 raft_consensus.cc:504] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } }
I20260806 09:21:22.175416 19836 leader_election.cc:288] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [CANDIDATE]: Term 1 election: Requested vote from peers d53f162523a64d2e941a0faa78c2b12d (127.16.86.131:36893), 12c6de8415a041d68420e8cad1fac790 (127.16.86.130:38871)
I20260806 09:21:22.175872 19732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb32ecc99ea44cc9acdd6c97721dc5d" candidate_uuid: "dab79d1f499840e59dd96ecd7cd9108b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53f162523a64d2e941a0faa78c2b12d"
I20260806 09:21:22.176008 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fcb32ecc99ea44cc9acdd6c97721dc5d" candidate_uuid: "dab79d1f499840e59dd96ecd7cd9108b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c6de8415a041d68420e8cad1fac790"
I20260806 09:21:22.176117 19732 raft_consensus.cc:3037] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:22.176309 19532 raft_consensus.cc:3037] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:22.178869 19732 raft_consensus.cc:2454] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab79d1f499840e59dd96ecd7cd9108b in term 1.
I20260806 09:21:22.179092 19532 raft_consensus.cc:2454] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dab79d1f499840e59dd96ecd7cd9108b in term 1.
I20260806 09:21:22.179603 19237 leader_election.cc:302] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c6de8415a041d68420e8cad1fac790, dab79d1f499840e59dd96ecd7cd9108b; no voters:
I20260806 09:21:22.179994 19836 raft_consensus.cc:2781] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:22.180637 19836 raft_consensus.cc:686] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 1 LEADER]: Becoming Leader. State: Replica: dab79d1f499840e59dd96ecd7cd9108b, State: Running, Role: LEADER
I20260806 09:21:22.181290 19836 consensus_queue.cc:237] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } }
I20260806 09:21:22.187155 19113 catalog_manager.cc:5495] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b reported cstate change: term changed from 0 to 1, leader changed from <none> to dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129). New cstate: current_term: 1 leader_uuid: "dab79d1f499840e59dd96ecd7cd9108b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } health_report { overall_health: UNKNOWN } } }
W20260806 09:21:22.212350 19800 tablet.cc:2369] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d: Can't schedule compaction. Clean time has not been advanced past its initial value.
Aug 06 09:21:22 dist-test-slave-h3ft krb5kdc[19050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008079, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:22.235484 19642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.218629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42054 (local address 127.16.86.190:37049)
0806 09:21:22.218899 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.218908 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.219705 (+ 797us) server_negotiation.cc:407] Connection header received
0806 09:21:22.219993 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.219999 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.220081 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.220202 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:22.221752 (+ 1550us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.222769 (+ 1017us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.224343 (+ 1574us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.224851 (+ 508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.224960 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.229385 (+ 4425us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:22.229390 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:22.229393 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:22.229434 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:22.232258 (+ 2824us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:22.233146 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:22.233150 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:22.233153 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:22.233287 (+ 134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:22.233698 (+ 411us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:22.233701 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:22.233703 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:22.233979 (+ 276us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:22.234101 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.234702 (+ 601us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.235078 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":107,"server-negotiator.queue_time_us":90}
I20260806 09:21:22.235491 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.218716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:37049 (local address 127.0.0.1:42054)
0806 09:21:22.219552 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.219573 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.219763 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.220292 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.220297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.221194 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:22.221529 (+ 335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.221540 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.223017 (+ 1477us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.223022 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.224172 (+ 1150us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.224190 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.225097 (+ 907us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.225102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.225306 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.226144 (+ 838us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:22.226165 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:22.229159 (+ 2994us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:22.232435 (+ 3276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:22.232447 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:22.232456 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:22.232982 (+ 526us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:22.233404 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:22.233407 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:22.233410 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:22.233582 (+ 172us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:22.234166 (+ 584us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:22.234174 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:22.234358 (+ 184us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.234532 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.234906 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":498,"threads_started":1}
I20260806 09:21:22.242273 19113 catalog_manager.cc:1908] Servicing CreateTable request from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:42054:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 09:21:22.243673 19113 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:21:22.252084 19309 tablet_service.cc:1467] Processing CreateTablet for tablet a5853e8a880747089758750baa00b7a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c28b9d08b3b94227bce3525743cd14a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:21:22.252015 19503 tablet_service.cc:1467] Processing CreateTablet for tablet a5853e8a880747089758750baa00b7a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c28b9d08b3b94227bce3525743cd14a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:21:22.252015 19695 tablet_service.cc:1467] Processing CreateTablet for tablet a5853e8a880747089758750baa00b7a9 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=c28b9d08b3b94227bce3525743cd14a0]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:21:22.253145 19309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5853e8a880747089758750baa00b7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:22.253275 19503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5853e8a880747089758750baa00b7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:22.253275 19695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a5853e8a880747089758750baa00b7a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:22.259552 19824 tablet_bootstrap.cc:492] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b: Bootstrap starting.
I20260806 09:21:22.261384 19824 tablet_bootstrap.cc:654] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:22.261404 19825 tablet_bootstrap.cc:492] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790: Bootstrap starting.
I20260806 09:21:22.262853 19821 tablet_bootstrap.cc:492] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: Bootstrap starting.
I20260806 09:21:22.263303 19825 tablet_bootstrap.cc:654] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:22.264147 19824 tablet_bootstrap.cc:492] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b: No bootstrap required, opened a new log
I20260806 09:21:22.264312 19824 ts_tablet_manager.cc:1397] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20260806 09:21:22.264914 19821 tablet_bootstrap.cc:654] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:22.265250 19824 raft_consensus.cc:348] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.265559 19824 raft_consensus.cc:374] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:22.265676 19824 raft_consensus.cc:729] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab79d1f499840e59dd96ecd7cd9108b, State: Initialized, Role: FOLLOWER
I20260806 09:21:22.265967 19825 tablet_bootstrap.cc:492] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790: No bootstrap required, opened a new log
I20260806 09:21:22.266144 19825 ts_tablet_manager.cc:1397] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 09:21:22.265978 19824 consensus_queue.cc:260] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.266894 19824 ts_tablet_manager.cc:1428] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260806 09:21:22.267053 19825 raft_consensus.cc:348] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.267335 19825 raft_consensus.cc:374] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:22.267450 19825 raft_consensus.cc:729] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c6de8415a041d68420e8cad1fac790, State: Initialized, Role: FOLLOWER
I20260806 09:21:22.267694 19836 tablet_replica.cc:439] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } }
I20260806 09:21:22.268100 19836 tablet_replica.cc:442] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:22.268281 19825 consensus_queue.cc:260] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.268807 19835 tablet_replica.cc:439] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } }
I20260806 09:21:22.269261 19835 tablet_replica.cc:442] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:22.269970 19825 ts_tablet_manager.cc:1428] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260806 09:21:22.272589 19821 tablet_bootstrap.cc:492] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: No bootstrap required, opened a new log
I20260806 09:21:22.272782 19821 ts_tablet_manager.cc:1397] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 09:21:22.273602 19821 raft_consensus.cc:348] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.273833 19821 raft_consensus.cc:374] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:22.273967 19821 raft_consensus.cc:729] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53f162523a64d2e941a0faa78c2b12d, State: Initialized, Role: FOLLOWER
I20260806 09:21:22.274272 19821 consensus_queue.cc:260] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.274947 19821 ts_tablet_manager.cc:1428] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 09:21:22.274887 19834 tablet_replica.cc:439] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } }
I20260806 09:21:22.275493 19834 tablet_replica.cc:442] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: This TxnStatusTablet replica's current role is: FOLLOWER
W20260806 09:21:22.288704 19602 tablet.cc:2369] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:21:22.288784 19834 raft_consensus.cc:482] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:21:22.289037 19834 raft_consensus.cc:504] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.290812 19834 leader_election.cc:288] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115), 12c6de8415a041d68420e8cad1fac790 (127.16.86.130:38871)
I20260806 09:21:22.298208 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.291356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:41749 (local address 127.16.86.129:33115)
0806 09:21:22.291646 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.291655 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.291780 (+ 125us) server_negotiation.cc:407] Connection header received
0806 09:21:22.292015 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.292021 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.292110 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.292189 (+ 79us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:22.292930 (+ 741us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.294157 (+ 1227us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.295916 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.296739 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.296925 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.297008 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.297797 (+ 789us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.297936 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 09:21:22.298588 19643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.291388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:33115 (local address 127.16.86.131:41749)
0806 09:21:22.291651 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.291674 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.291820 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.292361 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.292370 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.292396 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:22.292738 (+ 342us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.292752 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.294296 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.294303 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.295739 (+ 1436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.295752 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.296918 (+ 1166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.296924 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.297183 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.297219 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.297318 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.297493 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 09:21:22.299804 19349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5853e8a880747089758750baa00b7a9" candidate_uuid: "d53f162523a64d2e941a0faa78c2b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79d1f499840e59dd96ecd7cd9108b" is_pre_election: true
I20260806 09:21:22.300427 19819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.292198 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:52457 (local address 127.16.86.130:38871)
0806 09:21:22.292404 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.292410 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.293083 (+ 673us) server_negotiation.cc:407] Connection header received
0806 09:21:22.293288 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.293294 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.293363 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.293465 (+ 102us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:22.294138 (+ 673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.295342 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.298000 (+ 2658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.299323 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.299495 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.299567 (+ 72us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.299994 (+ 427us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.300124 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 09:21:22.300372 19349 raft_consensus.cc:2454] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d53f162523a64d2e941a0faa78c2b12d in term 0.
I20260806 09:21:22.300572 19857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.291900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:38871 (local address 127.16.86.131:52457)
0806 09:21:22.292942 (+ 1042us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.292962 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.293136 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.293519 (+ 383us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.293524 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.293552 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:22.293970 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.293982 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.295532 (+ 1550us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.295538 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.297804 (+ 2266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.297822 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.299594 (+ 1772us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.299600 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.299801 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.299844 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.299955 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.300123 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":622,"threads_started":1}
I20260806 09:21:22.301141 19631 leader_election.cc:302] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d53f162523a64d2e941a0faa78c2b12d, dab79d1f499840e59dd96ecd7cd9108b; no voters:
I20260806 09:21:22.301568 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5853e8a880747089758750baa00b7a9" candidate_uuid: "d53f162523a64d2e941a0faa78c2b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c6de8415a041d68420e8cad1fac790" is_pre_election: true
I20260806 09:21:22.301723 19834 raft_consensus.cc:2781] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:21:22.301901 19834 raft_consensus.cc:482] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:21:22.301914 19532 raft_consensus.cc:2454] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d53f162523a64d2e941a0faa78c2b12d in term 0.
I20260806 09:21:22.302002 19834 raft_consensus.cc:3037] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:22.304603 19834 raft_consensus.cc:504] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.305224 19834 leader_election.cc:288] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [CANDIDATE]: Term 1 election: Requested vote from peers dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115), 12c6de8415a041d68420e8cad1fac790 (127.16.86.130:38871)
I20260806 09:21:22.305891 19349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5853e8a880747089758750baa00b7a9" candidate_uuid: "d53f162523a64d2e941a0faa78c2b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79d1f499840e59dd96ecd7cd9108b"
I20260806 09:21:22.305969 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a5853e8a880747089758750baa00b7a9" candidate_uuid: "d53f162523a64d2e941a0faa78c2b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c6de8415a041d68420e8cad1fac790"
I20260806 09:21:22.306128 19349 raft_consensus.cc:3037] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:22.306316 19532 raft_consensus.cc:3037] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:22.308910 19532 raft_consensus.cc:2454] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d53f162523a64d2e941a0faa78c2b12d in term 1.
I20260806 09:21:22.309125 19349 raft_consensus.cc:2454] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d53f162523a64d2e941a0faa78c2b12d in term 1.
I20260806 09:21:22.309495 19631 leader_election.cc:302] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c6de8415a041d68420e8cad1fac790, d53f162523a64d2e941a0faa78c2b12d; no voters:
I20260806 09:21:22.309914 19834 raft_consensus.cc:2781] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:22.310491 19834 raft_consensus.cc:686] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [term 1 LEADER]: Becoming Leader. State: Replica: d53f162523a64d2e941a0faa78c2b12d, State: Running, Role: LEADER
I20260806 09:21:22.311170 19834 consensus_queue.cc:237] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.315372 19858 tablet_replica.cc:439] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: TxnStatusTablet state changed. Reason: New leader d53f162523a64d2e941a0faa78c2b12d. Latest consensus state: current_term: 1 leader_uuid: "d53f162523a64d2e941a0faa78c2b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } }
I20260806 09:21:22.315757 19858 tablet_replica.cc:442] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:21:22.316636 19861 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 09:21:22.316963 19861 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 09:21:22.317526 19114 catalog_manager.cc:5495] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d reported cstate change: term changed from 0 to 1, leader changed from <none> to d53f162523a64d2e941a0faa78c2b12d (127.16.86.131). New cstate: current_term: 1 leader_uuid: "d53f162523a64d2e941a0faa78c2b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } health_report { overall_health: HEALTHY } } }
I20260806 09:21:22.340432 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.332784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44460 (local address 127.16.86.131:36893)
0806 09:21:22.333415 (+ 631us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.333420 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.333436 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:21:22.333524 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.333528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.333631 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.333750 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:22.334337 (+ 587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.335515 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.337683 (+ 2168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.338205 (+ 522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.338316 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.338975 (+ 659us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:22.339751 (+ 776us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:22.339895 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.340050 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.340132 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":505}
I20260806 09:21:22.340538 19853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.332666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36893 (local address 127.0.0.1:44460)
0806 09:21:22.332992 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.333012 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.333141 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.333854 (+ 713us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.333860 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.333882 (+ 22us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:21:22.334125 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.334138 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.335706 (+ 1568us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.335711 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.337499 (+ 1788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.337515 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.338389 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.338396 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.338628 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.338725 (+ 97us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:21:22.339885 (+ 1160us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:21:22.339910 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.340017 (+ 107us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.340190 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 09:21:22.351190 19349 raft_consensus.cc:1261] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Refusing update from remote peer d53f162523a64d2e941a0faa78c2b12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:22.351214 19532 raft_consensus.cc:1261] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Refusing update from remote peer d53f162523a64d2e941a0faa78c2b12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:22.352334 19858 consensus_queue.cc:1035] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:22.352890 19834 consensus_queue.cc:1035] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:22.363173 19835 tablet_replica.cc:439] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790: TxnStatusTablet state changed. Reason: New leader d53f162523a64d2e941a0faa78c2b12d. Latest consensus state: current_term: 1 leader_uuid: "d53f162523a64d2e941a0faa78c2b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } }
I20260806 09:21:22.363814 19835 tablet_replica.cc:442] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:22.369257 19836 tablet_replica.cc:439] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b: TxnStatusTablet state changed. Reason: New leader d53f162523a64d2e941a0faa78c2b12d. Latest consensus state: current_term: 1 leader_uuid: "d53f162523a64d2e941a0faa78c2b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } }
I20260806 09:21:22.369679 19836 tablet_replica.cc:442] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:22.376487 19869 mvcc.cc:205] Tried to move back new op lower bound from 7315489105297702912 to 7315489105151127552. Current Snapshot: MvccSnapshot[applied={T|T < 7315489105297702912}]
I20260806 09:21:22.388077 19835 tablet_replica.cc:439] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d53f162523a64d2e941a0faa78c2b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } }
I20260806 09:21:22.388571 19835 tablet_replica.cc:442] T a5853e8a880747089758750baa00b7a9 P 12c6de8415a041d68420e8cad1fac790: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:22.391062 19836 tablet_replica.cc:439] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d53f162523a64d2e941a0faa78c2b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } }
I20260806 09:21:22.391431 19836 tablet_replica.cc:442] T a5853e8a880747089758750baa00b7a9 P dab79d1f499840e59dd96ecd7cd9108b: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:21:22.376006 19834 tablet_replica.cc:439] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d53f162523a64d2e941a0faa78c2b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } }
I20260806 09:21:22.392446 19834 tablet_replica.cc:442] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:21:22.393433 19834 tablet_replica.cc:439] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d53f162523a64d2e941a0faa78c2b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } } }
I20260806 09:21:22.393707 19834 tablet_replica.cc:442] T a5853e8a880747089758750baa00b7a9 P d53f162523a64d2e941a0faa78c2b12d: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:21:22.402334 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.392930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43304 (local address 127.16.86.129:33115)
0806 09:21:22.393258 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.393265 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.393284 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:21:22.393409 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.393415 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.393587 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.393779 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:22.394601 (+ 822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.395830 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.398183 (+ 2353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.398840 (+ 657us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.399007 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.399739 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:22.401536 (+ 1797us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:22.401683 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.401866 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.401962 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":191}
I20260806 09:21:22.423142 19887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.415724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36893 (local address 127.16.86.129:51299)
0806 09:21:22.416395 (+ 671us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.416414 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.416570 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.417038 (+ 468us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.417041 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.417051 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:21:22.417373 (+ 322us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.417404 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.418959 (+ 1555us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.418962 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.420181 (+ 1219us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.420193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.421383 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.421397 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.421525 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.421584 (+ 59us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:21:22.422187 (+ 603us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:21:22.422197 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.422406 (+ 209us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.422557 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":496,"spinlock_wait_cycles":2432,"thread_start_us":459,"threads_started":1}
I20260806 09:21:22.423111 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.415917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:51299 (local address 127.16.86.131:36893)
0806 09:21:22.416176 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.416185 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.416528 (+ 343us) server_negotiation.cc:407] Connection header received
0806 09:21:22.416758 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.416765 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.416886 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.417020 (+ 134us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:22.417560 (+ 540us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.418812 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.420333 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.421159 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.421313 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.421898 (+ 585us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:22.422073 (+ 175us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:22.422172 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.422467 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.422585 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:21:22.434861 19532 raft_consensus.cc:1261] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Refusing update from remote peer dab79d1f499840e59dd96ecd7cd9108b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:22.435297 19732 raft_consensus.cc:1261] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Refusing update from remote peer dab79d1f499840e59dd96ecd7cd9108b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:22.435778 19836 consensus_queue.cc:1035] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:22.436719 19836 consensus_queue.cc:1035] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:22.443826 19868 mvcc.cc:205] Tried to move back new op lower bound from 7315489105644965888 to 7315489104619696128. Current Snapshot: MvccSnapshot[applied={T|T < 7315489105644965888}]
I20260806 09:21:22.450333 19867 mvcc.cc:205] Tried to move back new op lower bound from 7315489105644965888 to 7315489104619696128. Current Snapshot: MvccSnapshot[applied={T|T < 7315489105644965888}]
I20260806 09:21:22.481491 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.474661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:35943 (local address 127.16.86.129:33115)
0806 09:21:22.474967 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.474978 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.475660 (+ 682us) server_negotiation.cc:407] Connection header received
0806 09:21:22.475913 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.475918 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.475990 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.476100 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:22.476659 (+ 559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.477518 (+ 859us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.479073 (+ 1555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.479636 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.479768 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.480772 (+ 1004us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:22.480890 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:22.480982 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.481173 (+ 191us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.481244 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 09:21:22.481875 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.474626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:33115 (local address 127.16.86.131:35943)
0806 09:21:22.475540 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.475564 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.475757 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.476114 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.476119 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.476130 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:21:22.476476 (+ 346us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.476485 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.477657 (+ 1172us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.477663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.478878 (+ 1215us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.478892 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.479983 (+ 1091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.479993 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.480261 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.480366 (+ 105us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:21:22.481016 (+ 650us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:21:22.481036 (+ 20us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.481141 (+ 105us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.481347 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":552,"threads_started":1}
I20260806 09:21:22.529124 19114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:22.529534 19114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:22.534883 19114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1: Sending DeleteTablet for 3 replicas of tablet fcb32ecc99ea44cc9acdd6c97721dc5d
I20260806 09:21:22.536375 19309 tablet_service.cc:1514] Processing DeleteTablet for tablet fcb32ecc99ea44cc9acdd6c97721dc5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:22 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:43296
I20260806 09:21:22.536552 19695 tablet_service.cc:1514] Processing DeleteTablet for tablet fcb32ecc99ea44cc9acdd6c97721dc5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:22 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:44450
I20260806 09:21:22.538967 19503 tablet_service.cc:1514] Processing DeleteTablet for tablet fcb32ecc99ea44cc9acdd6c97721dc5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:22 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:60902
I20260806 09:21:22.546393 19901 tablet_replica.cc:330] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b: stopping tablet replica
I20260806 09:21:22.546824 19114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:21:22.547838 19114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:21:22.547151 19901 raft_consensus.cc:2229] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:21:22.548800 19901 raft_consensus.cc:2258] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:22.551232 19901 ts_tablet_manager.cc:1905] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:22.563495 19694 tablet_service.cc:1467] Processing CreateTablet for tablet 566f8dde7e4e40b2921a3cc3ee270414 (DEFAULT_TABLE table=test-table [id=50eb69ec71bc41b4b2daa4d5ba42c3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:22.563808 19309 tablet_service.cc:1467] Processing CreateTablet for tablet 566f8dde7e4e40b2921a3cc3ee270414 (DEFAULT_TABLE table=test-table [id=50eb69ec71bc41b4b2daa4d5ba42c3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:22.563828 19500 tablet_service.cc:1467] Processing CreateTablet for tablet 566f8dde7e4e40b2921a3cc3ee270414 (DEFAULT_TABLE table=test-table [id=50eb69ec71bc41b4b2daa4d5ba42c3b3]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:22.564321 19309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566f8dde7e4e40b2921a3cc3ee270414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:22.564347 19694 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566f8dde7e4e40b2921a3cc3ee270414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:22.564396 19500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 566f8dde7e4e40b2921a3cc3ee270414. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:22.565145 19902 tablet_replica.cc:330] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d: stopping tablet replica
I20260806 09:21:22.565754 19902 raft_consensus.cc:2229] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:22.566198 19902 raft_consensus.cc:2258] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:22.567224 19903 tablet_replica.cc:330] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790: stopping tablet replica
I20260806 09:21:22.567765 19903 raft_consensus.cc:2229] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:22.568218 19903 raft_consensus.cc:2258] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:22.571059 19902 ts_tablet_manager.cc:1905] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:22.571135 19824 tablet_bootstrap.cc:492] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b: Bootstrap starting.
I20260806 09:21:22.572991 19824 tablet_bootstrap.cc:654] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:22.574294 19821 tablet_bootstrap.cc:492] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d: Bootstrap starting.
I20260806 09:21:22.576334 19821 tablet_bootstrap.cc:654] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:22.576874 19901 ts_tablet_manager.cc:1918] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:21:22.577207 19901 log.cc:1196] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/fcb32ecc99ea44cc9acdd6c97721dc5d
I20260806 09:21:22.578011 19901 ts_tablet_manager.cc:1939] T fcb32ecc99ea44cc9acdd6c97721dc5d P dab79d1f499840e59dd96ecd7cd9108b: Deleting consensus metadata
I20260806 09:21:22.580322 19093 catalog_manager.cc:4841] TS dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115): tablet fcb32ecc99ea44cc9acdd6c97721dc5d (table test-table [id=7c7365359c9242aa8387eaf1e89539f3]) successfully deleted
I20260806 09:21:22.584676 19825 tablet_bootstrap.cc:492] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790: Bootstrap starting.
I20260806 09:21:22.586906 19825 tablet_bootstrap.cc:654] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:22.586838 19902 ts_tablet_manager.cc:1918] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:21:22.587126 19902 log.cc:1196] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/fcb32ecc99ea44cc9acdd6c97721dc5d
I20260806 09:21:22.587812 19902 ts_tablet_manager.cc:1939] T fcb32ecc99ea44cc9acdd6c97721dc5d P d53f162523a64d2e941a0faa78c2b12d: Deleting consensus metadata
I20260806 09:21:22.592123 19095 catalog_manager.cc:4841] TS d53f162523a64d2e941a0faa78c2b12d (127.16.86.131:36893): tablet fcb32ecc99ea44cc9acdd6c97721dc5d (table test-table [id=7c7365359c9242aa8387eaf1e89539f3]) successfully deleted
I20260806 09:21:22.594815 19821 tablet_bootstrap.cc:492] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d: No bootstrap required, opened a new log
I20260806 09:21:22.595057 19821 ts_tablet_manager.cc:1397] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d: Time spent bootstrapping tablet: real 0.021s user 0.001s sys 0.010s
I20260806 09:21:22.595225 19824 tablet_bootstrap.cc:492] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b: No bootstrap required, opened a new log
I20260806 09:21:22.595448 19824 ts_tablet_manager.cc:1397] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b: Time spent bootstrapping tablet: real 0.024s user 0.011s sys 0.001s
I20260806 09:21:22.596284 19824 raft_consensus.cc:348] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.596551 19824 raft_consensus.cc:374] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:22.596628 19824 raft_consensus.cc:729] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab79d1f499840e59dd96ecd7cd9108b, State: Initialized, Role: FOLLOWER
I20260806 09:21:22.596655 19825 tablet_bootstrap.cc:492] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790: No bootstrap required, opened a new log
I20260806 09:21:22.596870 19825 ts_tablet_manager.cc:1397] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20260806 09:21:22.597083 19821 raft_consensus.cc:348] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.597420 19821 raft_consensus.cc:374] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:22.597535 19821 raft_consensus.cc:729] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53f162523a64d2e941a0faa78c2b12d, State: Initialized, Role: FOLLOWER
I20260806 09:21:22.597813 19821 consensus_queue.cc:260] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.597918 19825 raft_consensus.cc:348] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.598213 19825 raft_consensus.cc:374] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:22.598316 19825 raft_consensus.cc:729] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c6de8415a041d68420e8cad1fac790, State: Initialized, Role: FOLLOWER
I20260806 09:21:22.598549 19903 ts_tablet_manager.cc:1905] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:22.598706 19821 ts_tablet_manager.cc:1428] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 09:21:22.598699 19825 consensus_queue.cc:260] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.599646 19824 consensus_queue.cc:260] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.602900 19825 ts_tablet_manager.cc:1428] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260806 09:21:22.603801 19824 ts_tablet_manager.cc:1428] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260806 09:21:22.612587 19903 ts_tablet_manager.cc:1918] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:21:22.612912 19903 log.cc:1196] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/fcb32ecc99ea44cc9acdd6c97721dc5d
I20260806 09:21:22.613555 19903 ts_tablet_manager.cc:1939] T fcb32ecc99ea44cc9acdd6c97721dc5d P 12c6de8415a041d68420e8cad1fac790: Deleting consensus metadata
I20260806 09:21:22.615859 19093 catalog_manager.cc:4841] TS 12c6de8415a041d68420e8cad1fac790 (127.16.86.130:38871): tablet fcb32ecc99ea44cc9acdd6c97721dc5d (table test-table [id=7c7365359c9242aa8387eaf1e89539f3]) successfully deleted
I20260806 09:21:22.655524 19835 raft_consensus.cc:482] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:21:22.655860 19835 raft_consensus.cc:504] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.657976 19835 leader_election.cc:288] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115), d53f162523a64d2e941a0faa78c2b12d (127.16.86.131:36893)
I20260806 09:21:22.686254 19817 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.659036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:34367 (local address 127.16.86.129:33115)
0806 09:21:22.659409 (+ 373us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.659416 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.669741 (+ 10325us) server_negotiation.cc:407] Connection header received
0806 09:21:22.670379 (+ 638us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.670387 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.670482 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.670564 (+ 82us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:22.676554 (+ 5990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.677849 (+ 1295us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.680383 (+ 2534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.681925 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.682204 (+ 279us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.682310 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.685577 (+ 3267us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.685798 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":182}
I20260806 09:21:22.688344 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.659124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:50723 (local address 127.16.86.131:36893)
0806 09:21:22.659569 (+ 445us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.659575 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.670050 (+ 10475us) server_negotiation.cc:407] Connection header received
0806 09:21:22.670355 (+ 305us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.670361 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.670455 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.670524 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:22.672220 (+ 1696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.673572 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.683066 (+ 9494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.684595 (+ 1529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.684744 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.684844 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.687743 (+ 2899us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.687949 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314}
I20260806 09:21:22.689551 19909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.658595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36893 (local address 127.16.86.130:50723)
0806 09:21:22.669907 (+ 11312us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.669932 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.670182 (+ 250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.670771 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.670776 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.670802 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:22.671571 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.671588 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.673836 (+ 2248us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.673841 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.676300 (+ 2459us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.676317 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.685101 (+ 8784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.685108 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.685325 (+ 217us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.685365 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.688908 (+ 3543us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.689110 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11116,"mutex_wait_us":352,"thread_start_us":10473,"threads_started":1}
I20260806 09:21:22.690057 19883 raft_consensus.cc:482] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:21:22.690299 19883 raft_consensus.cc:504] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.691392 19883 leader_election.cc:288] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115), 12c6de8415a041d68420e8cad1fac790 (127.16.86.130:38871)
I20260806 09:21:22.692432 19349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566f8dde7e4e40b2921a3cc3ee270414" candidate_uuid: "d53f162523a64d2e941a0faa78c2b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79d1f499840e59dd96ecd7cd9108b" is_pre_election: true
I20260806 09:21:22.692587 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566f8dde7e4e40b2921a3cc3ee270414" candidate_uuid: "d53f162523a64d2e941a0faa78c2b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c6de8415a041d68420e8cad1fac790" is_pre_election: true
I20260806 09:21:22.692798 19349 raft_consensus.cc:2454] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d53f162523a64d2e941a0faa78c2b12d in term 0.
I20260806 09:21:22.692860 19532 raft_consensus.cc:2454] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d53f162523a64d2e941a0faa78c2b12d in term 0.
I20260806 09:21:22.693382 19631 leader_election.cc:302] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c6de8415a041d68420e8cad1fac790, d53f162523a64d2e941a0faa78c2b12d; no voters:
I20260806 09:21:22.694083 19732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566f8dde7e4e40b2921a3cc3ee270414" candidate_uuid: "12c6de8415a041d68420e8cad1fac790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53f162523a64d2e941a0faa78c2b12d" is_pre_election: true
I20260806 09:21:22.694337 19732 raft_consensus.cc:2454] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c6de8415a041d68420e8cad1fac790 in term 0.
I20260806 09:21:22.695115 19441 leader_election.cc:302] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c6de8415a041d68420e8cad1fac790, d53f162523a64d2e941a0faa78c2b12d; no voters:
I20260806 09:21:22.695626 19835 raft_consensus.cc:2781] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:21:22.695752 19835 raft_consensus.cc:482] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:21:22.695852 19835 raft_consensus.cc:3037] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:22.699514 19835 raft_consensus.cc:504] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.700582 19883 raft_consensus.cc:2781] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:21:22.700810 19883 raft_consensus.cc:482] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:21:22.700933 19883 raft_consensus.cc:3037] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:22.701681 19732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566f8dde7e4e40b2921a3cc3ee270414" candidate_uuid: "12c6de8415a041d68420e8cad1fac790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53f162523a64d2e941a0faa78c2b12d"
I20260806 09:21:22.702087 19835 leader_election.cc:288] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [CANDIDATE]: Term 1 election: Requested vote from peers dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115), d53f162523a64d2e941a0faa78c2b12d (127.16.86.131:36893)
I20260806 09:21:22.692517 19910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.658445 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:33115 (local address 127.16.86.130:34367)
0806 09:21:22.669524 (+ 11079us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.669565 (+ 41us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.670158 (+ 593us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.670810 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.670815 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.670847 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:22.671987 (+ 1140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.672000 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.678072 (+ 6072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.678078 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.680165 (+ 2087us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.680184 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.682550 (+ 2366us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.682557 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.682772 (+ 215us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.682813 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.686988 (+ 4175us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.687215 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10832,"mutex_wait_us":1,"thread_start_us":11315,"threads_started":1}
I20260806 09:21:22.703289 19349 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566f8dde7e4e40b2921a3cc3ee270414" candidate_uuid: "12c6de8415a041d68420e8cad1fac790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79d1f499840e59dd96ecd7cd9108b" is_pre_election: true
I20260806 09:21:22.703507 19347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566f8dde7e4e40b2921a3cc3ee270414" candidate_uuid: "12c6de8415a041d68420e8cad1fac790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79d1f499840e59dd96ecd7cd9108b"
I20260806 09:21:22.703646 19349 raft_consensus.cc:2454] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c6de8415a041d68420e8cad1fac790 in term 0.
I20260806 09:21:22.704473 19883 raft_consensus.cc:504] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.704943 19732 raft_consensus.cc:2379] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 12c6de8415a041d68420e8cad1fac790 in current term 1: Already voted for candidate d53f162523a64d2e941a0faa78c2b12d in this term.
I20260806 09:21:22.706257 19441 leader_election.cc:302] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 12c6de8415a041d68420e8cad1fac790; no voters: d53f162523a64d2e941a0faa78c2b12d, dab79d1f499840e59dd96ecd7cd9108b
I20260806 09:21:22.706799 19835 raft_consensus.cc:2726] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 09:21:22.707993 19347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566f8dde7e4e40b2921a3cc3ee270414" candidate_uuid: "d53f162523a64d2e941a0faa78c2b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79d1f499840e59dd96ecd7cd9108b"
I20260806 09:21:22.708312 19347 raft_consensus.cc:3037] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:22.708307 19532 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "566f8dde7e4e40b2921a3cc3ee270414" candidate_uuid: "d53f162523a64d2e941a0faa78c2b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c6de8415a041d68420e8cad1fac790"
I20260806 09:21:22.708895 19532 raft_consensus.cc:2379] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate d53f162523a64d2e941a0faa78c2b12d in current term 1: Already voted for candidate 12c6de8415a041d68420e8cad1fac790 in this term.
I20260806 09:21:22.709666 19883 leader_election.cc:288] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [CANDIDATE]: Term 1 election: Requested vote from peers dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115), 12c6de8415a041d68420e8cad1fac790 (127.16.86.130:38871)
I20260806 09:21:22.711916 19347 raft_consensus.cc:2454] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d53f162523a64d2e941a0faa78c2b12d in term 1.
I20260806 09:21:22.712481 19631 leader_election.cc:302] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d53f162523a64d2e941a0faa78c2b12d, dab79d1f499840e59dd96ecd7cd9108b; no voters: 12c6de8415a041d68420e8cad1fac790
I20260806 09:21:22.713266 19883 raft_consensus.cc:2781] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:22.713845 19883 raft_consensus.cc:686] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 1 LEADER]: Becoming Leader. State: Replica: d53f162523a64d2e941a0faa78c2b12d, State: Running, Role: LEADER
I20260806 09:21:22.714221 19883 consensus_queue.cc:237] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:22.719656 19114 catalog_manager.cc:5495] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d reported cstate change: term changed from 0 to 1, leader changed from <none> to d53f162523a64d2e941a0faa78c2b12d (127.16.86.131). New cstate: current_term: 1 leader_uuid: "d53f162523a64d2e941a0faa78c2b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } health_report { overall_health: HEALTHY } } }
I20260806 09:21:22.766181 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.758536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44462 (local address 127.16.86.131:36893)
0806 09:21:22.758756 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.758767 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.758791 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:21:22.758872 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.758877 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.758987 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.759175 (+ 188us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:22.759742 (+ 567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.761150 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.763299 (+ 2149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.764122 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.764293 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.764864 (+ 571us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:22.765049 (+ 185us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:22.765188 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.765700 (+ 512us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.765806 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:21:22.787887 19818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.780293 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:37375 (local address 127.16.86.131:36893)
0806 09:21:22.780516 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:22.780523 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:22.780544 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:21:22.780614 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:22.780619 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:22.780713 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:22.780823 (+ 110us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:22.781364 (+ 541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.782553 (+ 1189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.785085 (+ 2532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.785888 (+ 803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.786049 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.786665 (+ 616us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:22.786869 (+ 204us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:22.786981 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:22.787236 (+ 255us) server_negotiation.cc:299] Negotiation successful
0806 09:21:22.787443 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:21:22.787911 19895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:22.779406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36893 (local address 127.16.86.131:37375)
0806 09:21:22.779700 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:22.779732 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:22.779888 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:22.780898 (+ 1010us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:22.780904 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:22.780923 (+ 19us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:21:22.781170 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.781183 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.782740 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.782747 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.784877 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:22.784897 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:22.786092 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:22.786100 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:22.786355 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:22.786422 (+ 67us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:21:22.787036 (+ 614us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:21:22.787051 (+ 15us) client_negotiation.cc:769] Sending connection context
0806 09:21:22.787191 (+ 140us) client_negotiation.cc:240] Negotiation successful
0806 09:21:22.787366 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 09:21:22.803576 19532 raft_consensus.cc:1261] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Refusing update from remote peer d53f162523a64d2e941a0faa78c2b12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:22.803826 19347 raft_consensus.cc:1261] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Refusing update from remote peer d53f162523a64d2e941a0faa78c2b12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:22.804452 19877 consensus_queue.cc:1035] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:22.805231 19898 consensus_queue.cc:1035] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
W20260806 09:21:22.851233 19867 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 09:21:24.491834 19114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:24.492091 19114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42038:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:24.496966 19114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1: Sending DeleteTablet for 3 replicas of tablet 566f8dde7e4e40b2921a3cc3ee270414
I20260806 09:21:24.497885 19309 tablet_service.cc:1514] Processing DeleteTablet for tablet 566f8dde7e4e40b2921a3cc3ee270414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:24 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:43296
I20260806 09:21:24.498027 19695 tablet_service.cc:1514] Processing DeleteTablet for tablet 566f8dde7e4e40b2921a3cc3ee270414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:24 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:44450
I20260806 09:21:24.498095 19500 tablet_service.cc:1514] Processing DeleteTablet for tablet 566f8dde7e4e40b2921a3cc3ee270414 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:24 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:60902
I20260806 09:21:24.510031 19948 tablet_replica.cc:330] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790: stopping tablet replica
I20260806 09:21:24.510334 19946 tablet_replica.cc:330] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b: stopping tablet replica
I20260806 09:21:24.510341 19948 raft_consensus.cc:2229] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:24.510589 19946 raft_consensus.cc:2229] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:24.510651 19948 raft_consensus.cc:2258] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:24.510866 19946 raft_consensus.cc:2258] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:24.512534 19946 ts_tablet_manager.cc:1905] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:24.514231 19948 ts_tablet_manager.cc:1905] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:24.514494 19947 tablet_replica.cc:330] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d: stopping tablet replica
I20260806 09:21:24.514786 19947 raft_consensus.cc:2229] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:21:24.515409 19947 raft_consensus.cc:2258] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:24.516633 19947 ts_tablet_manager.cc:1905] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:24.520952 19946 ts_tablet_manager.cc:1918] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:21:24.521118 19946 log.cc:1196] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/566f8dde7e4e40b2921a3cc3ee270414
I20260806 09:21:24.521708 19946 ts_tablet_manager.cc:1939] T 566f8dde7e4e40b2921a3cc3ee270414 P dab79d1f499840e59dd96ecd7cd9108b: Deleting consensus metadata
I20260806 09:21:24.523216 19093 catalog_manager.cc:4841] TS dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115): tablet 566f8dde7e4e40b2921a3cc3ee270414 (table test-table [id=50eb69ec71bc41b4b2daa4d5ba42c3b3]) successfully deleted
I20260806 09:21:24.523304 19948 ts_tablet_manager.cc:1918] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:21:24.523456 19948 log.cc:1196] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/566f8dde7e4e40b2921a3cc3ee270414
I20260806 09:21:24.523900 19948 ts_tablet_manager.cc:1939] T 566f8dde7e4e40b2921a3cc3ee270414 P 12c6de8415a041d68420e8cad1fac790: Deleting consensus metadata
I20260806 09:21:24.524971 19947 ts_tablet_manager.cc:1918] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:21:24.525187 19947 log.cc:1196] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/566f8dde7e4e40b2921a3cc3ee270414
I20260806 09:21:24.525156 19093 catalog_manager.cc:4841] TS 12c6de8415a041d68420e8cad1fac790 (127.16.86.130:38871): tablet 566f8dde7e4e40b2921a3cc3ee270414 (table test-table [id=50eb69ec71bc41b4b2daa4d5ba42c3b3]) successfully deleted
I20260806 09:21:24.525775 19947 ts_tablet_manager.cc:1939] T 566f8dde7e4e40b2921a3cc3ee270414 P d53f162523a64d2e941a0faa78c2b12d: Deleting consensus metadata
I20260806 09:21:24.527173 19095 catalog_manager.cc:4841] TS d53f162523a64d2e941a0faa78c2b12d (127.16.86.131:36893): tablet 566f8dde7e4e40b2921a3cc3ee270414 (table test-table [id=50eb69ec71bc41b4b2daa4d5ba42c3b3]) successfully deleted
Aug 06 09:21:24 dist-test-slave-h3ft krb5kdc[19050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:21:24 dist-test-slave-h3ft krb5kdc[19050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:24.573372 19962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:24.556399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.16.86.190:37049)
0806 09:21:24.557235 (+ 836us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:24.557242 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:24.557271 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:21:24.557465 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:24.557470 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:24.557537 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:24.557660 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:24.559214 (+ 1554us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:24.560033 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:24.561365 (+ 1332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:24.561764 (+ 399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:24.561914 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:24.567429 (+ 5515us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:24.567434 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:24.567438 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:24.567475 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:24.570262 (+ 2787us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:24.571200 (+ 938us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:24.571204 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:24.571206 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:24.571360 (+ 154us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:24.571727 (+ 367us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:24.571730 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:24.571732 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:24.572083 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:24.572292 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:24.572848 (+ 556us) server_negotiation.cc:299] Negotiation successful
0806 09:21:24.573062 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":396,"threads_started":1}
I20260806 09:21:24.578290 19114 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:21:24.579198 19114 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:21:24.586859 19309 tablet_service.cc:1467] Processing CreateTablet for tablet 8b77a520783548c5856980e1c0db7161 (DEFAULT_TABLE table=test-table [id=1f564de574b94aef8adb1503977b5b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:24.587119 19695 tablet_service.cc:1467] Processing CreateTablet for tablet 8b77a520783548c5856980e1c0db7161 (DEFAULT_TABLE table=test-table [id=1f564de574b94aef8adb1503977b5b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:24.587137 19500 tablet_service.cc:1467] Processing CreateTablet for tablet 8b77a520783548c5856980e1c0db7161 (DEFAULT_TABLE table=test-table [id=1f564de574b94aef8adb1503977b5b94]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:24.587514 19309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b77a520783548c5856980e1c0db7161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:24.587594 19695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b77a520783548c5856980e1c0db7161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:24.587743 19500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 8b77a520783548c5856980e1c0db7161. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:24.593899 19963 tablet_bootstrap.cc:492] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790: Bootstrap starting.
I20260806 09:21:24.595332 19964 tablet_bootstrap.cc:492] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b: Bootstrap starting.
I20260806 09:21:24.595924 19963 tablet_bootstrap.cc:654] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:24.597322 19964 tablet_bootstrap.cc:654] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:24.599205 19965 tablet_bootstrap.cc:492] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d: Bootstrap starting.
I20260806 09:21:24.600728 19963 tablet_bootstrap.cc:492] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790: No bootstrap required, opened a new log
I20260806 09:21:24.600899 19963 ts_tablet_manager.cc:1397] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260806 09:21:24.600920 19965 tablet_bootstrap.cc:654] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:24.601853 19963 raft_consensus.cc:348] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.602183 19963 raft_consensus.cc:374] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:24.602309 19963 raft_consensus.cc:729] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c6de8415a041d68420e8cad1fac790, State: Initialized, Role: FOLLOWER
I20260806 09:21:24.602600 19963 consensus_queue.cc:260] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.602830 19964 tablet_bootstrap.cc:492] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b: No bootstrap required, opened a new log
I20260806 09:21:24.602980 19964 ts_tablet_manager.cc:1397] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.000s
I20260806 09:21:24.603569 19963 ts_tablet_manager.cc:1428] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260806 09:21:24.603873 19964 raft_consensus.cc:348] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.604106 19964 raft_consensus.cc:374] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:24.604203 19964 raft_consensus.cc:729] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab79d1f499840e59dd96ecd7cd9108b, State: Initialized, Role: FOLLOWER
I20260806 09:21:24.604478 19965 tablet_bootstrap.cc:492] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d: No bootstrap required, opened a new log
I20260806 09:21:24.604475 19964 consensus_queue.cc:260] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.604704 19965 ts_tablet_manager.cc:1397] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260806 09:21:24.605484 19964 ts_tablet_manager.cc:1428] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260806 09:21:24.605880 19965 raft_consensus.cc:348] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.606478 19965 raft_consensus.cc:374] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:24.606613 19965 raft_consensus.cc:729] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53f162523a64d2e941a0faa78c2b12d, State: Initialized, Role: FOLLOWER
I20260806 09:21:24.606982 19965 consensus_queue.cc:260] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.607746 19965 ts_tablet_manager.cc:1428] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 09:21:24.730350 19970 raft_consensus.cc:482] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:21:24.730621 19970 raft_consensus.cc:504] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.731496 19970 leader_election.cc:288] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115), d53f162523a64d2e941a0faa78c2b12d (127.16.86.131:36893)
I20260806 09:21:24.732342 19732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b77a520783548c5856980e1c0db7161" candidate_uuid: "12c6de8415a041d68420e8cad1fac790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53f162523a64d2e941a0faa78c2b12d" is_pre_election: true
I20260806 09:21:24.732425 19347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b77a520783548c5856980e1c0db7161" candidate_uuid: "12c6de8415a041d68420e8cad1fac790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79d1f499840e59dd96ecd7cd9108b" is_pre_election: true
I20260806 09:21:24.732765 19732 raft_consensus.cc:2454] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c6de8415a041d68420e8cad1fac790 in term 0.
I20260806 09:21:24.732769 19347 raft_consensus.cc:2454] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 12c6de8415a041d68420e8cad1fac790 in term 0.
I20260806 09:21:24.733433 19441 leader_election.cc:302] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12c6de8415a041d68420e8cad1fac790, d53f162523a64d2e941a0faa78c2b12d, dab79d1f499840e59dd96ecd7cd9108b; no voters:
I20260806 09:21:24.733796 19970 raft_consensus.cc:2781] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:21:24.733942 19970 raft_consensus.cc:482] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:21:24.734019 19970 raft_consensus.cc:3037] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:24.737286 19970 raft_consensus.cc:504] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.738085 19970 leader_election.cc:288] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [CANDIDATE]: Term 1 election: Requested vote from peers dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115), d53f162523a64d2e941a0faa78c2b12d (127.16.86.131:36893)
I20260806 09:21:24.738662 19347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b77a520783548c5856980e1c0db7161" candidate_uuid: "12c6de8415a041d68420e8cad1fac790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79d1f499840e59dd96ecd7cd9108b"
I20260806 09:21:24.738947 19347 raft_consensus.cc:3037] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:24.738848 19732 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "8b77a520783548c5856980e1c0db7161" candidate_uuid: "12c6de8415a041d68420e8cad1fac790" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d53f162523a64d2e941a0faa78c2b12d"
I20260806 09:21:24.739181 19732 raft_consensus.cc:3037] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:24.741605 19732 raft_consensus.cc:2454] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c6de8415a041d68420e8cad1fac790 in term 1.
I20260806 09:21:24.741626 19347 raft_consensus.cc:2454] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 12c6de8415a041d68420e8cad1fac790 in term 1.
I20260806 09:21:24.742313 19438 leader_election.cc:302] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 12c6de8415a041d68420e8cad1fac790, d53f162523a64d2e941a0faa78c2b12d, dab79d1f499840e59dd96ecd7cd9108b; no voters:
I20260806 09:21:24.742734 19970 raft_consensus.cc:2781] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:24.743418 19970 raft_consensus.cc:686] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 1 LEADER]: Becoming Leader. State: Replica: 12c6de8415a041d68420e8cad1fac790, State: Running, Role: LEADER
I20260806 09:21:24.743980 19970 consensus_queue.cc:237] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.748611 19114 catalog_manager.cc:5495] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 reported cstate change: term changed from 0 to 1, leader changed from <none> to 12c6de8415a041d68420e8cad1fac790 (127.16.86.130). New cstate: current_term: 1 leader_uuid: "12c6de8415a041d68420e8cad1fac790" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } health_report { overall_health: UNKNOWN } } }
I20260806 09:21:24.811138 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:24.804398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60904 (local address 127.16.86.130:38871)
0806 09:21:24.805245 (+ 847us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:24.805249 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:24.805277 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:21:24.805369 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:24.805372 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:24.805459 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:24.805674 (+ 215us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:24.806107 (+ 433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:24.806892 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:24.808476 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:24.808948 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:24.809055 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:24.809674 (+ 619us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:24.810330 (+ 656us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:24.810453 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:24.810663 (+ 210us) server_negotiation.cc:299] Negotiation successful
0806 09:21:24.810768 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":415,"threads_started":1}
I20260806 09:21:24.830892 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:24.824269 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36893 (local address 127.16.86.130:37451)
0806 09:21:24.825052 (+ 783us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:24.825082 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:24.825283 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:24.825773 (+ 490us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:24.825777 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:24.825790 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:21:24.826212 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:24.826222 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:24.827501 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:24.827506 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:24.828686 (+ 1180us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:24.828699 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:24.829478 (+ 779us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:24.829484 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:24.829620 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:24.829684 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:21:24.830159 (+ 475us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:21:24.830187 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 09:21:24.830273 (+ 86us) client_negotiation.cc:240] Negotiation successful
0806 09:21:24.830434 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":365,"threads_started":1}
I20260806 09:21:24.830866 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:24.824697 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:37451 (local address 127.16.86.131:36893)
0806 09:21:24.825429 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:24.825442 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:24.825464 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:21:24.825540 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:24.825544 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:24.825620 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:24.825769 (+ 149us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:24.826388 (+ 619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:24.827361 (+ 973us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:24.828829 (+ 1468us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:24.829256 (+ 427us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:24.829390 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:24.829926 (+ 536us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:24.830044 (+ 118us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:24.830143 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:24.830326 (+ 183us) server_negotiation.cc:299] Negotiation successful
0806 09:21:24.830413 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":356,"threads_started":1}
I20260806 09:21:24.841564 19347 raft_consensus.cc:1261] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Refusing update from remote peer 12c6de8415a041d68420e8cad1fac790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:24.841665 19732 raft_consensus.cc:1261] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Refusing update from remote peer 12c6de8415a041d68420e8cad1fac790: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:24.842550 19973 consensus_queue.cc:1035] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:24.843524 19973 consensus_queue.cc:1035] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:24.885090 19977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:24.878458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:41591 (local address 127.16.86.130:38871)
0806 09:21:24.878630 (+ 172us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:24.878638 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:24.878744 (+ 106us) server_negotiation.cc:407] Connection header received
0806 09:21:24.878960 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:24.878966 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:24.879098 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:24.879203 (+ 105us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:24.880076 (+ 873us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:24.880839 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:24.883348 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:24.883813 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:24.883923 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:24.884450 (+ 527us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:24.884556 (+ 106us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:24.884631 (+ 75us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:24.884791 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 09:21:24.884861 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:21:24.885499 19996 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:24.877864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:38871 (local address 127.16.86.131:41591)
0806 09:21:24.878595 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:24.878617 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:24.878798 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:24.879453 (+ 655us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:24.879459 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:24.879475 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:21:24.879840 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:24.879852 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:24.880988 (+ 1136us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:24.880995 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:24.882969 (+ 1974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:24.882988 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:24.884010 (+ 1022us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:24.884017 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:24.884220 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:24.884290 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:21:24.884658 (+ 368us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:21:24.884669 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 09:21:24.884753 (+ 84us) client_negotiation.cc:240] Negotiation successful
0806 09:21:24.884922 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":538,"thread_start_us":362,"threads_started":1}
I20260806 09:21:24.927906 19112 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:24.928272 19112 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:24.932806 19112 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1: Sending DeleteTablet for 3 replicas of tablet 8b77a520783548c5856980e1c0db7161
I20260806 09:21:24.934023 19309 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b77a520783548c5856980e1c0db7161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:24 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:43296
I20260806 09:21:24.934122 19695 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b77a520783548c5856980e1c0db7161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:24 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:44450
I20260806 09:21:24.934387 19946 tablet_replica.cc:330] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b: stopping tablet replica
I20260806 09:21:24.934643 19946 raft_consensus.cc:2229] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:24.934643 19500 tablet_service.cc:1514] Processing DeleteTablet for tablet 8b77a520783548c5856980e1c0db7161 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:24 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:60902
I20260806 09:21:24.934906 19946 raft_consensus.cc:2258] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:24.935074 19948 tablet_replica.cc:330] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790: stopping tablet replica
I20260806 09:21:24.935321 19947 tablet_replica.cc:330] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d: stopping tablet replica
I20260806 09:21:24.935382 19948 raft_consensus.cc:2229] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:21:24.935575 19947 raft_consensus.cc:2229] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:24.935807 19947 raft_consensus.cc:2258] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:24.936079 19948 raft_consensus.cc:2258] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:24.937395 19947 ts_tablet_manager.cc:1905] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:24.937577 19948 ts_tablet_manager.cc:1905] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:24.938570 19946 ts_tablet_manager.cc:1905] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:24.944936 19947 ts_tablet_manager.cc:1918] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:21:24.945151 19947 log.cc:1196] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/8b77a520783548c5856980e1c0db7161
I20260806 09:21:24.945935 19947 ts_tablet_manager.cc:1939] T 8b77a520783548c5856980e1c0db7161 P d53f162523a64d2e941a0faa78c2b12d: Deleting consensus metadata
I20260806 09:21:24.946599 19948 ts_tablet_manager.cc:1918] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:21:24.946899 19948 log.cc:1196] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/8b77a520783548c5856980e1c0db7161
I20260806 09:21:24.946530 19112 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42062:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 09:21:24.947425 19095 catalog_manager.cc:4841] TS d53f162523a64d2e941a0faa78c2b12d (127.16.86.131:36893): tablet 8b77a520783548c5856980e1c0db7161 (table test-table [id=1f564de574b94aef8adb1503977b5b94]) successfully deleted
W20260806 09:21:24.947672 19112 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:21:24.947892 19948 ts_tablet_manager.cc:1939] T 8b77a520783548c5856980e1c0db7161 P 12c6de8415a041d68420e8cad1fac790: Deleting consensus metadata
I20260806 09:21:24.949402 19093 catalog_manager.cc:4841] TS 12c6de8415a041d68420e8cad1fac790 (127.16.86.130:38871): tablet 8b77a520783548c5856980e1c0db7161 (table test-table [id=1f564de574b94aef8adb1503977b5b94]) successfully deleted
I20260806 09:21:24.950575 19946 ts_tablet_manager.cc:1918] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:21:24.950794 19946 log.cc:1196] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/8b77a520783548c5856980e1c0db7161
I20260806 09:21:24.951378 19946 ts_tablet_manager.cc:1939] T 8b77a520783548c5856980e1c0db7161 P dab79d1f499840e59dd96ecd7cd9108b: Deleting consensus metadata
I20260806 09:21:24.952821 19093 catalog_manager.cc:4841] TS dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115): tablet 8b77a520783548c5856980e1c0db7161 (table test-table [id=1f564de574b94aef8adb1503977b5b94]) successfully deleted
I20260806 09:21:24.957125 19309 tablet_service.cc:1467] Processing CreateTablet for tablet 4d50db8058bf41519b3ee31f8d6f5c49 (DEFAULT_TABLE table=test-table [id=919a3a92db054b08bfe6469837d0ad44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:24.957717 19309 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d50db8058bf41519b3ee31f8d6f5c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:24.957779 19500 tablet_service.cc:1467] Processing CreateTablet for tablet 4d50db8058bf41519b3ee31f8d6f5c49 (DEFAULT_TABLE table=test-table [id=919a3a92db054b08bfe6469837d0ad44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:24.958405 19500 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d50db8058bf41519b3ee31f8d6f5c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:24.964854 19964 tablet_bootstrap.cc:492] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b: Bootstrap starting.
I20260806 09:21:24.966080 19695 tablet_service.cc:1467] Processing CreateTablet for tablet 4d50db8058bf41519b3ee31f8d6f5c49 (DEFAULT_TABLE table=test-table [id=919a3a92db054b08bfe6469837d0ad44]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:24.966495 19963 tablet_bootstrap.cc:492] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790: Bootstrap starting.
I20260806 09:21:24.966650 19695 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 4d50db8058bf41519b3ee31f8d6f5c49. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:24.966717 19964 tablet_bootstrap.cc:654] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:24.968621 19963 tablet_bootstrap.cc:654] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:24.976106 19965 tablet_bootstrap.cc:492] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d: Bootstrap starting.
I20260806 09:21:24.976795 19963 tablet_bootstrap.cc:492] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790: No bootstrap required, opened a new log
I20260806 09:21:24.977033 19963 ts_tablet_manager.cc:1397] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.004s
I20260806 09:21:24.977933 19963 raft_consensus.cc:348] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.978164 19965 tablet_bootstrap.cc:654] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:24.978233 19963 raft_consensus.cc:374] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:24.978330 19963 raft_consensus.cc:729] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12c6de8415a041d68420e8cad1fac790, State: Initialized, Role: FOLLOWER
I20260806 09:21:24.978609 19963 consensus_queue.cc:260] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.982370 19963 ts_tablet_manager.cc:1428] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260806 09:21:24.992084 19965 tablet_bootstrap.cc:492] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d: No bootstrap required, opened a new log
I20260806 09:21:24.992352 19965 ts_tablet_manager.cc:1397] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.003s
I20260806 09:21:24.994248 19964 tablet_bootstrap.cc:492] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b: No bootstrap required, opened a new log
I20260806 09:21:24.994493 19964 ts_tablet_manager.cc:1397] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b: Time spent bootstrapping tablet: real 0.030s user 0.016s sys 0.002s
I20260806 09:21:24.994621 19965 raft_consensus.cc:348] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.994941 19965 raft_consensus.cc:374] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:24.995074 19965 raft_consensus.cc:729] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d53f162523a64d2e941a0faa78c2b12d, State: Initialized, Role: FOLLOWER
I20260806 09:21:24.995399 19965 consensus_queue.cc:260] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.995383 19964 raft_consensus.cc:348] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.995659 19964 raft_consensus.cc:374] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:24.995757 19964 raft_consensus.cc:729] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dab79d1f499840e59dd96ecd7cd9108b, State: Initialized, Role: FOLLOWER
I20260806 09:21:24.996016 19964 consensus_queue.cc:260] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:24.996162 19965 ts_tablet_manager.cc:1428] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260806 09:21:24.996727 19964 ts_tablet_manager.cc:1428] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260806 09:21:25.186542 19926 raft_consensus.cc:482] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:21:25.186870 19926 raft_consensus.cc:504] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:25.187778 19926 leader_election.cc:288] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115), 12c6de8415a041d68420e8cad1fac790 (127.16.86.130:38871)
I20260806 09:21:25.188514 19529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d50db8058bf41519b3ee31f8d6f5c49" candidate_uuid: "d53f162523a64d2e941a0faa78c2b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c6de8415a041d68420e8cad1fac790" is_pre_election: true
I20260806 09:21:25.188666 19347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d50db8058bf41519b3ee31f8d6f5c49" candidate_uuid: "d53f162523a64d2e941a0faa78c2b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79d1f499840e59dd96ecd7cd9108b" is_pre_election: true
I20260806 09:21:25.188830 19529 raft_consensus.cc:2454] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d53f162523a64d2e941a0faa78c2b12d in term 0.
I20260806 09:21:25.189128 19347 raft_consensus.cc:2454] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d53f162523a64d2e941a0faa78c2b12d in term 0.
I20260806 09:21:25.189538 19631 leader_election.cc:302] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 12c6de8415a041d68420e8cad1fac790, d53f162523a64d2e941a0faa78c2b12d; no voters:
I20260806 09:21:25.189946 19926 raft_consensus.cc:2781] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:21:25.190132 19926 raft_consensus.cc:482] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:21:25.190265 19926 raft_consensus.cc:3037] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:25.194352 19926 raft_consensus.cc:504] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:25.195231 19926 leader_election.cc:288] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [CANDIDATE]: Term 1 election: Requested vote from peers dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115), 12c6de8415a041d68420e8cad1fac790 (127.16.86.130:38871)
I20260806 09:21:25.196051 19347 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d50db8058bf41519b3ee31f8d6f5c49" candidate_uuid: "d53f162523a64d2e941a0faa78c2b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dab79d1f499840e59dd96ecd7cd9108b"
I20260806 09:21:25.196082 19529 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "4d50db8058bf41519b3ee31f8d6f5c49" candidate_uuid: "d53f162523a64d2e941a0faa78c2b12d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12c6de8415a041d68420e8cad1fac790"
I20260806 09:21:25.196364 19347 raft_consensus.cc:3037] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:25.196347 19529 raft_consensus.cc:3037] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:25.199605 19347 raft_consensus.cc:2454] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d53f162523a64d2e941a0faa78c2b12d in term 1.
I20260806 09:21:25.200163 19529 raft_consensus.cc:2454] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d53f162523a64d2e941a0faa78c2b12d in term 1.
I20260806 09:21:25.200261 19631 leader_election.cc:302] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d53f162523a64d2e941a0faa78c2b12d, dab79d1f499840e59dd96ecd7cd9108b; no voters:
I20260806 09:21:25.200680 19926 raft_consensus.cc:2781] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:25.200899 19926 raft_consensus.cc:686] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 1 LEADER]: Becoming Leader. State: Replica: d53f162523a64d2e941a0faa78c2b12d, State: Running, Role: LEADER
I20260806 09:21:25.201269 19926 consensus_queue.cc:237] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } }
I20260806 09:21:25.205494 19114 catalog_manager.cc:5495] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d reported cstate change: term changed from 0 to 1, leader changed from <none> to d53f162523a64d2e941a0faa78c2b12d (127.16.86.131). New cstate: current_term: 1 leader_uuid: "d53f162523a64d2e941a0faa78c2b12d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d53f162523a64d2e941a0faa78c2b12d" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36893 } health_report { overall_health: HEALTHY } } }
I20260806 09:21:25.234836 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:25.229878 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44470 (local address 127.16.86.131:36893)
0806 09:21:25.230069 (+ 191us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:25.230075 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:25.230093 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:21:25.230269 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:25.230275 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:25.230357 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:25.230461 (+ 104us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:25.230876 (+ 415us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:25.231717 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:25.233039 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:25.233455 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:25.233590 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:25.234015 (+ 425us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:25.234131 (+ 116us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:25.234254 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:25.234454 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 09:21:25.234531 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 09:21:25.247339 19529 raft_consensus.cc:1261] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Refusing update from remote peer d53f162523a64d2e941a0faa78c2b12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:25.247557 19347 raft_consensus.cc:1261] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Refusing update from remote peer d53f162523a64d2e941a0faa78c2b12d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:25.248176 19984 consensus_queue.cc:1035] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "12c6de8415a041d68420e8cad1fac790" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 38871 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:25.248536 19926 consensus_queue.cc:1035] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [LEADER]: Connected to new peer: Peer: permanent_uuid: "dab79d1f499840e59dd96ecd7cd9108b" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33115 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 09:21:25.275905 19918 tablet_replica.cc:1298] Aborted: operation has been aborted: cancelling pending write operations
I20260806 09:21:26.905828 19114 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:26.906133 19114 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:42062:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:26.911275 19114 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00f28b1f810f4b33b8ad559ab85206a1: Sending DeleteTablet for 3 replicas of tablet 4d50db8058bf41519b3ee31f8d6f5c49
I20260806 09:21:26.912420 19309 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d50db8058bf41519b3ee31f8d6f5c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:26 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:43296
I20260806 09:21:26.912526 19500 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d50db8058bf41519b3ee31f8d6f5c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:26 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:60902
I20260806 09:21:26.912587 19694 tablet_service.cc:1514] Processing DeleteTablet for tablet 4d50db8058bf41519b3ee31f8d6f5c49 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:26 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:44450
I20260806 09:21:26.913905 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 19214
I20260806 09:21:26.924975 20038 tablet_replica.cc:330] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b: stopping tablet replica
I20260806 09:21:26.925457 20038 raft_consensus.cc:2229] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:26.925889 20038 raft_consensus.cc:2258] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:26.927798 20038 ts_tablet_manager.cc:1905] T 4d50db8058bf41519b3ee31f8d6f5c49 P dab79d1f499840e59dd96ecd7cd9108b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:26.928519 20040 tablet_replica.cc:330] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d: stopping tablet replica
I20260806 09:21:26.928810 20040 raft_consensus.cc:2229] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:21:26.928829 20039 tablet_replica.cc:330] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790: stopping tablet replica
I20260806 09:21:26.929075 20039 raft_consensus.cc:2229] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:26.929308 20040 raft_consensus.cc:2258] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:26.929366 20039 raft_consensus.cc:2258] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:26.930696 20039 ts_tablet_manager.cc:1905] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:26.930900 20040 ts_tablet_manager.cc:1905] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 09:21:26 dist-test-slave-h3ft krb5kdc[19050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:21:26.952981 20043 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:26.916013 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43318 (local address 127.16.86.129:33115)
0806 09:21:26.934797 (+ 18784us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:26.934801 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:26.934832 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:21:26.934892 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:26.934897 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:26.934965 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:26.935173 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:26.937044 (+ 1871us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:26.937869 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:26.939464 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:26.939960 (+ 496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:26.940083 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:26.946144 (+ 6061us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:26.946168 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:26.946185 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:26.946225 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:26.949738 (+ 3513us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:26.950658 (+ 920us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:26.950666 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:26.950669 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:26.950814 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:26.951282 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:26.951288 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:26.951292 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:26.951681 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:26.952012 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:26.952330 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 09:21:26.952583 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18648,"thread_start_us":19018,"threads_started":1}
W20260806 09:21:26.954005 19401 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:43318
W20260806 09:21:26.954970 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 19214: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:21:26.955354 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 19214
I20260806 09:21:26.970456 20039 ts_tablet_manager.cc:1918] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:21:26.970664 20039 log.cc:1196] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/4d50db8058bf41519b3ee31f8d6f5c49
I20260806 09:21:26.971246 20039 ts_tablet_manager.cc:1939] T 4d50db8058bf41519b3ee31f8d6f5c49 P 12c6de8415a041d68420e8cad1fac790: Deleting consensus metadata
W20260806 09:21:27.005900 19093 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.86.129:33115 (error 108)
I20260806 09:21:27.007135 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 19412
I20260806 09:21:27.009704 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:27.007986 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:33115 (local address 127.0.0.1:43326)
0806 09:21:27.009274 (+ 1288us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:27.009476 (+ 202us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1105,"thread_start_us":906,"threads_started":1}
W20260806 09:21:27.009945 19093 catalog_manager.cc:4584] TS dab79d1f499840e59dd96ecd7cd9108b (127.16.86.129:33115): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 4d50db8058bf41519b3ee31f8d6f5c49: Network error: Client connection negotiation failed: client connection to 127.16.86.129:33115: connect: Connection refused (error 111)
Aug 06 09:21:27 dist-test-slave-h3ft krb5kdc[19050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:21:27.026926 20048 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:27.008062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60910 (local address 127.16.86.130:38871)
0806 09:21:27.009574 (+ 1512us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:27.009582 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:27.009607 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:21:27.009689 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:27.009695 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:27.009791 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:27.010526 (+ 735us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:27.012120 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:27.013003 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:27.014265 (+ 1262us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:27.014689 (+ 424us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:27.014864 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:27.019460 (+ 4596us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:27.019484 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:27.019508 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:27.019565 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:27.023175 (+ 3610us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:27.024138 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:27.024157 (+ 19us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:27.024163 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:27.024509 (+ 346us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:27.024977 (+ 468us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:27.024981 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:27.024983 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:27.025555 (+ 572us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:27.025875 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:27.026194 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 09:21:27.026501 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1350,"thread_start_us":1194,"threads_started":1}
W20260806 09:21:27.027851 19594 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:60910
W20260806 09:21:27.028657 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 19412: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:21:27.028952 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 19412
I20260806 09:21:27.064286 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:27.063686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:33115 (local address 127.0.0.1:43328)
0806 09:21:27.063912 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:27.064043 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 09:21:27.064463 20040 ts_tablet_manager.cc:1918] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20260806 09:21:27.064661 20040 log.cc:1196] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/4d50db8058bf41519b3ee31f8d6f5c49
I20260806 09:21:27.065214 20040 ts_tablet_manager.cc:1939] T 4d50db8058bf41519b3ee31f8d6f5c49 P d53f162523a64d2e941a0faa78c2b12d: Deleting consensus metadata
I20260806 09:21:27.065866 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:27.065420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:33115 (local address 127.0.0.1:43336)
0806 09:21:27.065646 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:27.065734 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20260806 09:21:27.068876 19095 catalog_manager.cc:4841] TS d53f162523a64d2e941a0faa78c2b12d (127.16.86.131:36893): tablet 4d50db8058bf41519b3ee31f8d6f5c49 (table test-table [id=919a3a92db054b08bfe6469837d0ad44]) successfully deleted
I20260806 09:21:27.076047 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 19607
I20260806 09:21:27.077010 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:27.076536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:38871 (local address 127.0.0.1:60918)
0806 09:21:27.076725 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:27.076819 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.130:38871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
Aug 06 09:21:27 dist-test-slave-h3ft krb5kdc[19050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008084, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:21:27.095306 20049 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:27.077144 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44478 (local address 127.16.86.131:36893)
0806 09:21:27.077752 (+ 608us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:27.077757 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:27.077776 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:21:27.077842 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:27.077845 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:27.077912 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:27.078027 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:27.079907 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:27.080787 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:27.082308 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:27.082781 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:27.082893 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:27.088639 (+ 5746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:27.088657 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:27.088675 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:27.088715 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:27.092031 (+ 3316us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:27.092967 (+ 936us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:27.092974 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:27.092977 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:27.093158 (+ 181us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:27.093645 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:27.093649 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:27.093652 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:27.094041 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:27.094319 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:27.094701 (+ 382us) server_negotiation.cc:299] Negotiation successful
0806 09:21:27.094894 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":312,"threads_started":1}
W20260806 09:21:27.096025 19791 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:44478
W20260806 09:21:27.096885 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 19607: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:21:27.097106 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 19607
I20260806 09:21:27.110035 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:27.109346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:38871 (local address 127.0.0.1:60926)
0806 09:21:27.109729 (+ 383us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:27.109840 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.130:38871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":246}
I20260806 09:21:27.111454 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:27.111044 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:38871 (local address 127.0.0.1:60932)
0806 09:21:27.111285 (+ 241us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:27.111352 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.130:38871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 09:21:27.127560 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:27.126764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:33115 (local address 127.0.0.1:43340)
0806 09:21:27.126987 (+ 223us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:27.127308 (+ 321us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 09:21:27.128605 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:27.128249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:33115 (local address 127.0.0.1:43352)
0806 09:21:27.128419 (+ 170us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:27.128502 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20260806 09:21:27.141036 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 19069
I20260806 09:21:27.142756 19170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:27.297562 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:27.296088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:38871 (local address 127.0.0.1:60938)
0806 09:21:27.297047 (+ 959us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:27.297225 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.130:38871: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":791}
I20260806 09:21:27.298252 20046 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:27.297431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:33115 (local address 127.0.0.1:43354)
0806 09:21:27.297903 (+ 472us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:27.298054 (+ 151us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:33115: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":607,"threads_started":1}
I20260806 09:21:27.298249 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 19069
2026-08-06T09:21:27Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9520 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:21:27 dist-test-slave-h3ft krb5kdc[20058](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:21:27 dist-test-slave-h3ft krb5kdc[20058](info): set up 2 sockets
Aug 06 09:21:27 dist-test-slave-h3ft krb5kdc[20058](info): commencing operation
krb5kdc: starting...
W20260806 09:21:29.022568 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.666s user 0.000s sys 0.030s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:21:29 dist-test-slave-h3ft krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:21:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:21:29.307924 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:37131
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:37131
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:29.711272 20076 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:29.711686 20076 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:29.711790 20076 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:29.719862 20076 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:29.720021 20076 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:29.720074 20076 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:29.720131 20076 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:29.720191 20076 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:21:29.731900 20076 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:37131
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:37131
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:29.733682 20076 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:29.735387 20076 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:29.750958 20082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:29.750970 20083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:29.750867 20085 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:29.751362 20076 server_base.cc:1034] running on GCE node
I20260806 09:21:29.752985 20076 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:29.754805 20076 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:29.756085 20076 hybrid_clock.cc:648] HybridClock initialized: now 1786008089756015 us; error 59 us; skew 500 ppm
I20260806 09:21:29.756558 20076 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:29 dist-test-slave-h3ft krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008089, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:29.760499 20076 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:21:29.762404 20076 webserver.cc:460] Webserver started at http://127.16.86.190:41403/ using document root <none> and password file <none>
I20260806 09:21:29.763332 20076 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:29.763458 20076 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:29.763789 20076 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:29.767500 20076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "1f57da15bc0d48f6bfc0a54ca71b2a44"
format_stamp: "Formatted at 2026-08-06 09:21:29 on dist-test-slave-h3ft"
server_key: "6c08582228f19be043ecf6f7a257ebc4"
server_key_iv: "673f8c30e17d13cf3acd506e58c128ba"
server_key_version: "encryptionkey@0"
I20260806 09:21:29.768375 20076 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "1f57da15bc0d48f6bfc0a54ca71b2a44"
format_stamp: "Formatted at 2026-08-06 09:21:29 on dist-test-slave-h3ft"
server_key: "6c08582228f19be043ecf6f7a257ebc4"
server_key_iv: "673f8c30e17d13cf3acd506e58c128ba"
server_key_version: "encryptionkey@0"
I20260806 09:21:29.775190 20076 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 09:21:29.779807 20094 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:29.781747 20076 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:21:29.782025 20076 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "1f57da15bc0d48f6bfc0a54ca71b2a44"
format_stamp: "Formatted at 2026-08-06 09:21:29 on dist-test-slave-h3ft"
server_key: "6c08582228f19be043ecf6f7a257ebc4"
server_key_iv: "673f8c30e17d13cf3acd506e58c128ba"
server_key_version: "encryptionkey@0"
I20260806 09:21:29.782310 20076 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:29.809900 20076 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:29.815819 20076 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:29.816179 20076 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:29.841034 20076 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:37131
I20260806 09:21:29.840986 20170 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:37131 every 8 connection(s)
I20260806 09:21:29.843531 20076 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:29.848378 20172 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:29.849421 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 20076
I20260806 09:21:29.849730 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:29.850493 16730 external_mini_cluster.cc:1428] Setting key 4622720802dbb1ca69c6dcdd887dc1ee
I20260806 09:21:29.865743 20172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44: Bootstrap starting.
Aug 06 09:21:29 dist-test-slave-h3ft krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008089, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:29.870131 20172 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:29.871603 20172 log.cc:824] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:29.875813 20172 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44: No bootstrap required, opened a new log
I20260806 09:21:29.880916 20177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:29.853124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33736 (local address 127.16.86.190:37131)
0806 09:21:29.854157 (+ 1033us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:29.854174 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:29.854233 (+ 59us) server_negotiation.cc:407] Connection header received
0806 09:21:29.856312 (+ 2079us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:29.856346 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:29.856886 (+ 540us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:29.857338 (+ 452us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:29.859546 (+ 2208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:29.861062 (+ 1516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:29.863231 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:29.864177 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:29.864380 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:29.870451 (+ 6071us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:29.870471 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:29.870500 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:29.870565 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:29.875171 (+ 4606us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:29.876205 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:29.876217 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:29.876226 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:29.876570 (+ 344us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:29.877029 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:29.877035 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:29.877038 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:29.877876 (+ 838us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:29.878307 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:29.878668 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 09:21:29.879291 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":758,"thread_start_us":479,"threads_started":1}
I20260806 09:21:29.886281 20172 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f57da15bc0d48f6bfc0a54ca71b2a44" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37131 } }
I20260806 09:21:29.886744 20172 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:29.886884 20172 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1f57da15bc0d48f6bfc0a54ca71b2a44, State: Initialized, Role: FOLLOWER
I20260806 09:21:29.887585 20172 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f57da15bc0d48f6bfc0a54ca71b2a44" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37131 } }
I20260806 09:21:29.887846 20172 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:29.888005 20172 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:29.888227 20172 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:29.890691 20172 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f57da15bc0d48f6bfc0a54ca71b2a44" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37131 } }
I20260806 09:21:29.891290 20172 leader_election.cc:302] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1f57da15bc0d48f6bfc0a54ca71b2a44; no voters:
I20260806 09:21:29.892019 20172 leader_election.cc:288] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:29.892233 20179 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:29.893224 20179 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [term 1 LEADER]: Becoming Leader. State: Replica: 1f57da15bc0d48f6bfc0a54ca71b2a44, State: Running, Role: LEADER
I20260806 09:21:29.893911 20179 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f57da15bc0d48f6bfc0a54ca71b2a44" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37131 } }
I20260806 09:21:29.894718 20172 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:29.897981 20182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1f57da15bc0d48f6bfc0a54ca71b2a44. Latest consensus state: current_term: 1 leader_uuid: "1f57da15bc0d48f6bfc0a54ca71b2a44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f57da15bc0d48f6bfc0a54ca71b2a44" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37131 } } }
I20260806 09:21:29.898595 20182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:29.902486 20181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1f57da15bc0d48f6bfc0a54ca71b2a44" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1f57da15bc0d48f6bfc0a54ca71b2a44" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37131 } } }
I20260806 09:21:29.902884 20189 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:29.902935 20181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:29.909963 20189 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:29.922250 20189 catalog_manager.cc:1347] Generated new cluster ID: 5bf42f556edc43a6a5f9bed641b1e759
I20260806 09:21:29.922464 20189 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:29.953006 20189 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:29.954757 20189 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:29.969736 20189 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1f57da15bc0d48f6bfc0a54ca71b2a44: Generated new TSK 0
I20260806 09:21:29.971067 20189 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:21:30.060631 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:37131
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:30.501959 20213 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:30.502331 20213 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:30.502440 20213 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:30.510705 20213 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:30.510864 20213 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:30.510994 20213 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:21:30.521955 20213 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:37131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:30.523775 20213 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:30.525426 20213 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:30.542143 20225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:30.542718 20224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:30.543036 20227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:30.543669 20213 server_base.cc:1034] running on GCE node
I20260806 09:21:30.544543 20213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:30.545784 20213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:30.547158 20213 hybrid_clock.cc:648] HybridClock initialized: now 1786008090547068 us; error 55 us; skew 500 ppm
I20260806 09:21:30.547577 20213 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:30 dist-test-slave-h3ft krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008090, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:30.551430 20213 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:21:30.553228 20213 webserver.cc:460] Webserver started at http://127.16.86.129:40713/ using document root <none> and password file <none>
I20260806 09:21:30.554018 20213 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:30.554183 20213 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:30.554534 20213 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:30.558007 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "b8f464f0e8ac403e9da4f445ee1a24d0"
format_stamp: "Formatted at 2026-08-06 09:21:30 on dist-test-slave-h3ft"
server_key: "a258e577f5236c557d962dacbe39411e"
server_key_iv: "d923b27ac5b2dca5fe6d4b192eb00d0b"
server_key_version: "encryptionkey@0"
I20260806 09:21:30.558866 20213 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "b8f464f0e8ac403e9da4f445ee1a24d0"
format_stamp: "Formatted at 2026-08-06 09:21:30 on dist-test-slave-h3ft"
server_key: "a258e577f5236c557d962dacbe39411e"
server_key_iv: "d923b27ac5b2dca5fe6d4b192eb00d0b"
server_key_version: "encryptionkey@0"
I20260806 09:21:30.564903 20213 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260806 09:21:30.569355 20236 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:30.571336 20213 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:21:30.571547 20213 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "b8f464f0e8ac403e9da4f445ee1a24d0"
format_stamp: "Formatted at 2026-08-06 09:21:30 on dist-test-slave-h3ft"
server_key: "a258e577f5236c557d962dacbe39411e"
server_key_iv: "d923b27ac5b2dca5fe6d4b192eb00d0b"
server_key_version: "encryptionkey@0"
I20260806 09:21:30.571781 20213 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:30.594792 20213 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:30.599844 20213 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:30.600178 20213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:30.601294 20213 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:30.602841 20213 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:30.602952 20213 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:30.603137 20213 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:30.603214 20213 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:30.644647 20213 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:38631
I20260806 09:21:30.644838 20390 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:38631 every 8 connection(s)
I20260806 09:21:30.646189 20213 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:21:30.655745 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 20213
I20260806 09:21:30.656054 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:21:30.656816 16730 external_mini_cluster.cc:1428] Setting key 8872cf5ddf09467f57bc078694136b34
Aug 06 09:21:30 dist-test-slave-h3ft krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008090, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:30.674175 20393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:30.649864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:55325 (local address 127.16.86.190:37131)
0806 09:21:30.650482 (+ 618us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:30.650487 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:30.651973 (+ 1486us) server_negotiation.cc:407] Connection header received
0806 09:21:30.654325 (+ 2352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:30.654331 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:30.654457 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:30.654614 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:30.656904 (+ 2290us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:30.657936 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:30.660115 (+ 2179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:30.660614 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:30.660743 (+ 129us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:30.666421 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:30.666428 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:30.666432 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:30.666477 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:30.669636 (+ 3159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:30.670910 (+ 1274us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:30.670919 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:30.670926 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:30.671151 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:30.671746 (+ 595us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:30.671754 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:30.671760 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:30.672217 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:30.672413 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:30.673316 (+ 903us) server_negotiation.cc:299] Negotiation successful
0806 09:21:30.673639 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":477,"thread_start_us":320,"threads_started":1}
I20260806 09:21:30.674903 20395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:30.650625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:37131 (local address 127.16.86.129:55325)
0806 09:21:30.651730 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:30.651794 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:30.653973 (+ 2179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:30.654908 (+ 935us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:30.654918 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:30.655812 (+ 894us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:30.656702 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:30.656725 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:30.658166 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:30.658171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:30.659917 (+ 1746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:30.659935 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:30.660934 (+ 999us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:30.660945 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:30.661361 (+ 416us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:30.662372 (+ 1011us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:30.662422 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:30.666061 (+ 3639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:30.669859 (+ 3798us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:30.669872 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:30.669895 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:30.670682 (+ 787us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:30.671297 (+ 615us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:30.671302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:30.671305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:30.671524 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:30.672375 (+ 851us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:30.672382 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:30.672845 (+ 463us) client_negotiation.cc:769] Sending connection context
0806 09:21:30.673132 (+ 287us) client_negotiation.cc:240] Negotiation successful
0806 09:21:30.673583 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":766,"thread_start_us":467,"threads_started":1}
I20260806 09:21:30.677211 20391 heartbeater.cc:347] Connected to a master server at 127.16.86.190:37131
I20260806 09:21:30.677716 20391 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:30.678875 20391 heartbeater.cc:511] Master 127.16.86.190:37131 requested a full tablet report, sending...
I20260806 09:21:30.682015 20121 ts_manager.cc:194] Registered new tserver with Master: b8f464f0e8ac403e9da4f445ee1a24d0 (127.16.86.129:38631)
I20260806 09:21:30.683745 20121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:55325
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:21:30.725067 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:37131
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:31.156085 20400 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:31.156463 20400 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:31.156718 20400 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:31.165156 20400 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:31.165325 20400 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:31.165525 20400 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:21:31.177472 20400 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:37131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:31.179152 20400 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:31.180706 20400 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:31.197636 20409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:31.198469 20413 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:31.198489 20410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:31.199175 20400 server_base.cc:1034] running on GCE node
I20260806 09:21:31.199962 20400 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:31.201175 20400 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:31.202430 20400 hybrid_clock.cc:648] HybridClock initialized: now 1786008091202373 us; error 44 us; skew 500 ppm
I20260806 09:21:31.202852 20400 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:31 dist-test-slave-h3ft krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008091, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:31.213372 20400 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:21:31.215541 20400 webserver.cc:460] Webserver started at http://127.16.86.130:45269/ using document root <none> and password file <none>
I20260806 09:21:31.216399 20400 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:31.216578 20400 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:31.216949 20400 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:31.220352 20400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "9972d500a2ab4bf4a984ff35995e7c54"
format_stamp: "Formatted at 2026-08-06 09:21:31 on dist-test-slave-h3ft"
server_key: "aa2957c89f5f2da1a2279992eaa3a884"
server_key_iv: "7691018367d6bac6fe0955d28ec2d78c"
server_key_version: "encryptionkey@0"
I20260806 09:21:31.221267 20400 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "9972d500a2ab4bf4a984ff35995e7c54"
format_stamp: "Formatted at 2026-08-06 09:21:31 on dist-test-slave-h3ft"
server_key: "aa2957c89f5f2da1a2279992eaa3a884"
server_key_iv: "7691018367d6bac6fe0955d28ec2d78c"
server_key_version: "encryptionkey@0"
I20260806 09:21:31.227793 20400 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260806 09:21:31.232360 20421 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:31.234099 20400 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:21:31.234328 20400 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "9972d500a2ab4bf4a984ff35995e7c54"
format_stamp: "Formatted at 2026-08-06 09:21:31 on dist-test-slave-h3ft"
server_key: "aa2957c89f5f2da1a2279992eaa3a884"
server_key_iv: "7691018367d6bac6fe0955d28ec2d78c"
server_key_version: "encryptionkey@0"
I20260806 09:21:31.234633 20400 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:31.256361 20400 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:31.261641 20400 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:31.261996 20400 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:31.263257 20400 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:31.264981 20400 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:31.265105 20400 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:31.265291 20400 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:31.265381 20400 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:31.308537 20400 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:46005
I20260806 09:21:31.308650 20595 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:46005 every 8 connection(s)
I20260806 09:21:31.309948 20400 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:21:31.318279 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 20400
I20260806 09:21:31.318506 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:21:31.318979 16730 external_mini_cluster.cc:1428] Setting key 80037de2b575078b880db3b8c08982ae
Aug 06 09:21:31 dist-test-slave-h3ft krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008091, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:31.336759 20599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:31.313419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:43115 (local address 127.16.86.190:37131)
0806 09:21:31.314006 (+ 587us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:31.314013 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:31.315120 (+ 1107us) server_negotiation.cc:407] Connection header received
0806 09:21:31.317415 (+ 2295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:31.317427 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:31.317537 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:31.317672 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:31.319764 (+ 2092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:31.320863 (+ 1099us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:31.322928 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:31.323824 (+ 896us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:31.324013 (+ 189us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:31.328416 (+ 4403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:31.328425 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:31.328434 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:31.328489 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:31.332516 (+ 4027us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:31.333665 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:31.333669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:31.333672 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:31.333820 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:31.334413 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:31.334417 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:31.334419 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:31.334808 (+ 389us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:31.334961 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:31.336049 (+ 1088us) server_negotiation.cc:299] Negotiation successful
0806 09:21:31.336399 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":4,"server-negotiator.queue_time_us":438,"thread_start_us":273,"threads_started":1}
I20260806 09:21:31.338117 20600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:31.313996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:37131 (local address 127.16.86.130:43115)
0806 09:21:31.314844 (+ 848us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:31.314901 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:31.317019 (+ 2118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:31.317981 (+ 962us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:31.317991 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:31.318741 (+ 750us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:31.319574 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:31.319595 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:31.321127 (+ 1532us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:31.321137 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:31.322742 (+ 1605us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:31.322760 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:31.324146 (+ 1386us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:31.324153 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:31.324361 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:31.325228 (+ 867us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:31.325255 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:31.328120 (+ 2865us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:31.332717 (+ 4597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:31.332731 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:31.332757 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:31.333485 (+ 728us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:31.333951 (+ 466us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:31.333958 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:31.333965 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:31.334223 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:31.334978 (+ 755us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:31.334989 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:31.335532 (+ 543us) client_negotiation.cc:769] Sending connection context
0806 09:21:31.335888 (+ 356us) client_negotiation.cc:240] Negotiation successful
0806 09:21:31.336490 (+ 602us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":556,"thread_start_us":321,"threads_started":1}
I20260806 09:21:31.340226 20597 heartbeater.cc:347] Connected to a master server at 127.16.86.190:37131
I20260806 09:21:31.340687 20597 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:31.341638 20597 heartbeater.cc:511] Master 127.16.86.190:37131 requested a full tablet report, sending...
I20260806 09:21:31.343694 20121 ts_manager.cc:194] Registered new tserver with Master: 9972d500a2ab4bf4a984ff35995e7c54 (127.16.86.130:46005)
I20260806 09:21:31.344712 20121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:43115
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:21:31.384815 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:37131
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:39365
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:21:31.688298 20391 heartbeater.cc:503] Master 127.16.86.190:37131 was elected leader, sending a full tablet report...
W20260806 09:21:31.758889 20603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:31.759275 20603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:31.759378 20603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:31.766731 20603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:31.766860 20603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:31.766973 20603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:21:31.777204 20603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:39365
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:37131
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:31.778524 20603 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:31.779929 20603 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:31.794991 20610 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:31.794885 20613 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:31.794943 20611 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:31.796211 20603 server_base.cc:1034] running on GCE node
I20260806 09:21:31.796952 20603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:31.797938 20603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:31.799183 20603 hybrid_clock.cc:648] HybridClock initialized: now 1786008091799090 us; error 72 us; skew 500 ppm
I20260806 09:21:31.799638 20603 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:31 dist-test-slave-h3ft krb5kdc[20058](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008091, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:31.803630 20603 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:21:31.805503 20603 webserver.cc:460] Webserver started at http://127.16.86.131:40449/ using document root <none> and password file <none>
I20260806 09:21:31.806282 20603 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:31.806438 20603 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:31.806780 20603 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:31.810107 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "f1a7ee52238d47559441afa59f1e1c7b"
format_stamp: "Formatted at 2026-08-06 09:21:31 on dist-test-slave-h3ft"
server_key: "2a7cefaa109091b1609064306f3d9114"
server_key_iv: "a0c98f2c4729c4f76bc0cebee56b75b6"
server_key_version: "encryptionkey@0"
I20260806 09:21:31.810945 20603 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "f1a7ee52238d47559441afa59f1e1c7b"
format_stamp: "Formatted at 2026-08-06 09:21:31 on dist-test-slave-h3ft"
server_key: "2a7cefaa109091b1609064306f3d9114"
server_key_iv: "a0c98f2c4729c4f76bc0cebee56b75b6"
server_key_version: "encryptionkey@0"
I20260806 09:21:31.816957 20603 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 09:21:31.820999 20622 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:31.822649 20603 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 09:21:31.822882 20603 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "f1a7ee52238d47559441afa59f1e1c7b"
format_stamp: "Formatted at 2026-08-06 09:21:31 on dist-test-slave-h3ft"
server_key: "2a7cefaa109091b1609064306f3d9114"
server_key_iv: "a0c98f2c4729c4f76bc0cebee56b75b6"
server_key_version: "encryptionkey@0"
I20260806 09:21:31.823189 20603 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:31.836186 20603 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:31.841216 20603 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:31.841545 20603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:31.842736 20603 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:31.844396 20603 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:31.844478 20603 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:31.844619 20603 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:31.844674 20603 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:31.889227 20603 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:35891
I20260806 09:21:31.889281 20777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:35891 every 8 connection(s)
I20260806 09:21:31.890584 20603 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:21:31.892066 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 20603
I20260806 09:21:31.892318 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:21:31.893040 16730 external_mini_cluster.cc:1428] Setting key 0056c5803ababb9b4aba4e1a4517bb3e
Aug 06 09:21:31 dist-test-slave-h3ft krb5kdc[20058](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008091, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:31.916056 20780 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:31.894900 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:34711 (local address 127.16.86.190:37131)
0806 09:21:31.895579 (+ 679us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:31.895588 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:31.896248 (+ 660us) server_negotiation.cc:407] Connection header received
0806 09:21:31.898368 (+ 2120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:31.898374 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:31.898486 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:31.898603 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:31.900298 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:31.901176 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:31.902778 (+ 1602us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:31.903340 (+ 562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:31.903475 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:31.907932 (+ 4457us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:31.907938 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:31.907941 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:31.907995 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:31.912185 (+ 4190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:31.913180 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:31.913186 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:31.913190 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:31.913348 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:31.913851 (+ 503us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:31.913859 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:31.913862 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:31.914189 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:31.914383 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:31.915337 (+ 954us) server_negotiation.cc:299] Negotiation successful
0806 09:21:31.915602 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":339,"threads_started":1}
I20260806 09:21:31.917303 20782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:31.894973 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:37131 (local address 127.16.86.131:34711)
0806 09:21:31.896034 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:31.896093 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:31.898071 (+ 1978us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:31.898876 (+ 805us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:31.898894 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:31.899504 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:31.900097 (+ 593us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:31.900117 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:31.901352 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:31.901356 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:31.902611 (+ 1255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:31.902629 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:31.903637 (+ 1008us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:31.903644 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:31.903959 (+ 315us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:31.904793 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:31.904818 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:31.907576 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:31.912397 (+ 4821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:31.912408 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:31.912428 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:31.913028 (+ 600us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:31.913476 (+ 448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:31.913487 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:31.913490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:31.913716 (+ 226us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:31.914334 (+ 618us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:31.914344 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:31.914808 (+ 464us) client_negotiation.cc:769] Sending connection context
0806 09:21:31.915211 (+ 403us) client_negotiation.cc:240] Negotiation successful
0806 09:21:31.915911 (+ 700us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":386,"threads_started":1}
I20260806 09:21:31.919088 20778 heartbeater.cc:347] Connected to a master server at 127.16.86.190:37131
I20260806 09:21:31.919461 20778 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:31.920318 20778 heartbeater.cc:511] Master 127.16.86.190:37131 requested a full tablet report, sending...
I20260806 09:21:31.922205 20121 ts_manager.cc:194] Registered new tserver with Master: f1a7ee52238d47559441afa59f1e1c7b (127.16.86.131:35891)
I20260806 09:21:31.923365 20121 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:34711
I20260806 09:21:31.924299 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 09:21:31.976369 20780 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:21:31.972986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33750 (local address 127.16.86.190:37131)
0806 09:21:31.973192 (+ 206us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:31.973197 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:31.973615 (+ 418us) server_negotiation.cc:407] Connection header received
0806 09:21:31.973971 (+ 356us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:31.973976 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:31.974095 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:31.974225 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:31.976066 (+ 1841us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33750: BlockingRecv error: recv got EOF from 127.0.0.1:33750 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:21:31.980983 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 20213
W20260806 09:21:31.984813 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 20213: Not authorized: Client connection negotiation failed: client connection to 127.16.86.129:38631: server requires authentication, but client does not have Kerberos credentials available
I20260806 09:21:31.985082 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 20213
I20260806 09:21:32.013710 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 20400
W20260806 09:21:32.017496 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 20400: Not authorized: Client connection negotiation failed: client connection to 127.16.86.130:46005: server requires authentication, but client does not have Kerberos credentials available
I20260806 09:21:32.017729 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 20400
I20260806 09:21:32.043286 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 20603
W20260806 09:21:32.047215 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 20603: Not authorized: Client connection negotiation failed: client connection to 127.16.86.131:35891: server requires authentication, but client does not have Kerberos credentials available
I20260806 09:21:32.047461 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 20603
W20260806 09:21:32.047586 20795 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:21:32.044809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41354 (local address 127.16.86.131:35891)
0806 09:21:32.045583 (+ 774us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:32.045590 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:32.045627 (+ 37us) server_negotiation.cc:407] Connection header received
0806 09:21:32.045702 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:32.045709 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:32.045930 (+ 221us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:32.046090 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:32.047148 (+ 1058us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:41354: BlockingRecv error: recv got EOF from 127.0.0.1:41354 (error 108)
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":439,"threads_started":1}
I20260806 09:21:32.073695 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 20076
I20260806 09:21:32.075359 20165 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:32.183513 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 20076
2026-08-06T09:21:32Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (4877 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:21:32 dist-test-slave-h3ft krb5kdc[20801](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:21:32 dist-test-slave-h3ft krb5kdc[20801](info): set up 2 sockets
Aug 06 09:21:32 dist-test-slave-h3ft krb5kdc[20801](info): commencing operation
krb5kdc: starting...
W20260806 09:21:33.533630 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.298s user 0.004s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:21:33 dist-test-slave-h3ft krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:21:33Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:21:33.817013 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:33423
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:46085
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:33423
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:34.167461 20819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:34.167779 20819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:34.167938 20819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:34.175300 20819 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:34.175423 20819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:34.175498 20819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:34.175623 20819 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:34.175698 20819 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:21:34.185593 20819 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46085
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:33423
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:33423
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:34.186931 20819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:34.188310 20819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:34.202006 20824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:34.202153 20827 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:34.202153 20825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:34.203413 20819 server_base.cc:1034] running on GCE node
I20260806 09:21:34.204358 20819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:34.205931 20819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:34.207198 20819 hybrid_clock.cc:648] HybridClock initialized: now 1786008094207137 us; error 45 us; skew 500 ppm
I20260806 09:21:34.207631 20819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:34 dist-test-slave-h3ft krb5kdc[20801](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008094, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:34.211777 20819 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:21:34.213698 20819 webserver.cc:460] Webserver started at http://127.16.86.190:33269/ using document root <none> and password file <none>
I20260806 09:21:34.214452 20819 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:34.214603 20819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:34.214949 20819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:34.218101 20819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "c4b937a233284399a5fcb9ff474b9f35"
format_stamp: "Formatted at 2026-08-06 09:21:34 on dist-test-slave-h3ft"
server_key: "8705eca9fc23df9cd15869e9119a81db"
server_key_iv: "7042db7486f2674615c56f1ea58d6b94"
server_key_version: "encryptionkey@0"
I20260806 09:21:34.218924 20819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "c4b937a233284399a5fcb9ff474b9f35"
format_stamp: "Formatted at 2026-08-06 09:21:34 on dist-test-slave-h3ft"
server_key: "8705eca9fc23df9cd15869e9119a81db"
server_key_iv: "7042db7486f2674615c56f1ea58d6b94"
server_key_version: "encryptionkey@0"
I20260806 09:21:34.224728 20819 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260806 09:21:34.228785 20835 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:34.230378 20819 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:21:34.230671 20819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "c4b937a233284399a5fcb9ff474b9f35"
format_stamp: "Formatted at 2026-08-06 09:21:34 on dist-test-slave-h3ft"
server_key: "8705eca9fc23df9cd15869e9119a81db"
server_key_iv: "7042db7486f2674615c56f1ea58d6b94"
server_key_version: "encryptionkey@0"
I20260806 09:21:34.230902 20819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:34.274303 20819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:34.279232 20819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:34.279541 20819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:34.301745 20819 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:33423
I20260806 09:21:34.301821 20904 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:33423 every 8 connection(s)
I20260806 09:21:34.303341 20819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:34.307200 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 20819
I20260806 09:21:34.307469 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:34.308456 16730 external_mini_cluster.cc:1428] Setting key ad2fc683d609f5b6fb7243c33bb0abf1
I20260806 09:21:34.308941 20906 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:21:34 dist-test-slave-h3ft krb5kdc[20801](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008093, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:34.327173 20906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35: Bootstrap starting.
I20260806 09:21:34.331943 20906 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:34.333679 20906 log.cc:824] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:34.337425 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:34.310867 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.16.86.190:33423)
0806 09:21:34.311883 (+ 1016us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:34.311896 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:34.311946 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:21:34.314654 (+ 2708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:34.314679 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:34.315207 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:34.315590 (+ 383us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:34.317857 (+ 2267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:34.319353 (+ 1496us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:34.321294 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:34.322026 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:34.322183 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:34.328103 (+ 5920us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:34.328112 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:34.328130 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:34.328175 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:34.331877 (+ 3702us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:34.332976 (+ 1099us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:34.332983 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:34.332991 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:34.333182 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:34.333672 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:34.333675 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:34.333677 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:34.334730 (+ 1053us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:34.335124 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:34.335844 (+ 720us) server_negotiation.cc:299] Negotiation successful
0806 09:21:34.336208 (+ 364us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":772,"thread_start_us":447,"threads_started":1}
I20260806 09:21:34.338129 20906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35: No bootstrap required, opened a new log
I20260806 09:21:34.342665 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:21:34.352702 20906 raft_consensus.cc:348] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b937a233284399a5fcb9ff474b9f35" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33423 } }
I20260806 09:21:34.353525 20906 raft_consensus.cc:374] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:34.353668 20906 raft_consensus.cc:729] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4b937a233284399a5fcb9ff474b9f35, State: Initialized, Role: FOLLOWER
I20260806 09:21:34.354676 20906 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b937a233284399a5fcb9ff474b9f35" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33423 } }
I20260806 09:21:34.355113 20906 raft_consensus.cc:388] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:34.355301 20906 raft_consensus.cc:482] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:34.355538 20906 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:34.358919 20906 raft_consensus.cc:504] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b937a233284399a5fcb9ff474b9f35" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33423 } }
I20260806 09:21:34.359757 20906 leader_election.cc:302] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c4b937a233284399a5fcb9ff474b9f35; no voters:
I20260806 09:21:34.360674 20906 leader_election.cc:288] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:34.360837 20913 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:34.362636 20913 raft_consensus.cc:686] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [term 1 LEADER]: Becoming Leader. State: Replica: c4b937a233284399a5fcb9ff474b9f35, State: Running, Role: LEADER
I20260806 09:21:34.363510 20913 consensus_queue.cc:237] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b937a233284399a5fcb9ff474b9f35" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33423 } }
I20260806 09:21:34.364243 20906 sys_catalog.cc:564] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:34.367219 20916 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c4b937a233284399a5fcb9ff474b9f35. Latest consensus state: current_term: 1 leader_uuid: "c4b937a233284399a5fcb9ff474b9f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b937a233284399a5fcb9ff474b9f35" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33423 } } }
I20260806 09:21:34.367826 20916 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:34.368623 20914 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c4b937a233284399a5fcb9ff474b9f35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c4b937a233284399a5fcb9ff474b9f35" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33423 } } }
I20260806 09:21:34.368995 20914 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:34.373075 20927 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:34.384977 20927 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:34.411250 20927 catalog_manager.cc:1347] Generated new cluster ID: f3871ec070004d87900d14c5d9df4b4b
I20260806 09:21:34.411435 20927 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:34.441221 20927 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:34.444623 20927 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:34.462956 20927 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c4b937a233284399a5fcb9ff474b9f35: Generated new TSK 0
I20260806 09:21:34.464747 20927 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:21:34.467923 20909 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:34.443811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38896 (local address 127.16.86.190:33423)
0806 09:21:34.444040 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:34.444049 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:34.455877 (+ 11828us) server_negotiation.cc:407] Connection header received
0806 09:21:34.456558 (+ 681us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:34.456565 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:34.456675 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:34.456766 (+ 91us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:34.459749 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:34.461119 (+ 1370us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:34.463051 (+ 1932us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:34.463875 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:34.464027 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:34.465899 (+ 1872us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:34.465907 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:34.465914 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:34.466921 (+ 1007us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:34.466985 (+ 64us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:34.467049 (+ 64us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:34.467168 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:34.467399 (+ 231us) server_negotiation.cc:299] Negotiation successful
0806 09:21:34.467526 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:21:34.484570 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 20819
I20260806 09:21:34.486162 20899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:34.597275 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 20819
2026-08-06T09:21:34Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2414 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:21:34 dist-test-slave-h3ft krb5kdc[20955](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:21:34 dist-test-slave-h3ft krb5kdc[20955](info): set up 2 sockets
Aug 06 09:21:34 dist-test-slave-h3ft krb5kdc[20955](info): commencing operation
krb5kdc: starting...
W20260806 09:21:35.935969 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.286s user 0.002s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:21:36 dist-test-slave-h3ft krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:21:36Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:36Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:21:36.228505 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:38007
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:38007
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:36.634222 20975 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:36.634608 20975 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:36.634709 20975 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:36.642858 20975 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:36.643034 20975 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:36.643092 20975 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:36.643150 20975 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:36.643204 20975 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:21:36.654696 20975 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:38007
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:38007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:36.656626 20975 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:36.658289 20975 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:36.672739 20980 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:36.673134 20982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:36.673254 20984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:36.674690 20975 server_base.cc:1034] running on GCE node
I20260806 09:21:36.675998 20975 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:36.678083 20975 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:36.679426 20975 hybrid_clock.cc:648] HybridClock initialized: now 1786008096679330 us; error 55 us; skew 500 ppm
I20260806 09:21:36.680001 20975 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:36 dist-test-slave-h3ft krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008096, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:36.689069 20975 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:21:36.691495 20975 webserver.cc:460] Webserver started at http://127.16.86.190:34695/ using document root <none> and password file <none>
I20260806 09:21:36.692523 20975 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:36.692684 20975 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:36.693095 20975 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:36.697224 20975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "40c7b8cfc287438a84d4e08db673a4ab"
format_stamp: "Formatted at 2026-08-06 09:21:36 on dist-test-slave-h3ft"
server_key: "1dfee72111a0a36899b4029652afc170"
server_key_iv: "2dd57c6e04beafcd30c2f649b98e6859"
server_key_version: "encryptionkey@0"
I20260806 09:21:36.698282 20975 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "40c7b8cfc287438a84d4e08db673a4ab"
format_stamp: "Formatted at 2026-08-06 09:21:36 on dist-test-slave-h3ft"
server_key: "1dfee72111a0a36899b4029652afc170"
server_key_iv: "2dd57c6e04beafcd30c2f649b98e6859"
server_key_version: "encryptionkey@0"
I20260806 09:21:36.705726 20975 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260806 09:21:36.711215 20993 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:36.713443 20975 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 09:21:36.713781 20975 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "40c7b8cfc287438a84d4e08db673a4ab"
format_stamp: "Formatted at 2026-08-06 09:21:36 on dist-test-slave-h3ft"
server_key: "1dfee72111a0a36899b4029652afc170"
server_key_iv: "2dd57c6e04beafcd30c2f649b98e6859"
server_key_version: "encryptionkey@0"
I20260806 09:21:36.714126 20975 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:36.762204 20975 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:36.768173 20975 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:36.768579 20975 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:36.793660 20975 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:38007
I20260806 09:21:36.793784 21059 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:38007 every 8 connection(s)
I20260806 09:21:36.795605 20975 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:36.800275 21060 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:36.800454 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 20975
I20260806 09:21:36.800686 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:36.801599 16730 external_mini_cluster.cc:1428] Setting key 37d4cd0b3b8a8942b39e28bc7885eb5a
I20260806 09:21:36.817150 21060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab: Bootstrap starting.
Aug 06 09:21:36 dist-test-slave-h3ft krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008096, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:36.821844 21060 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:36.823150 21060 log.cc:824] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:36.827282 21060 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab: No bootstrap required, opened a new log
I20260806 09:21:36.832319 21064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:36.804451 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46008 (local address 127.16.86.190:38007)
0806 09:21:36.805605 (+ 1154us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:36.805626 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:36.805686 (+ 60us) server_negotiation.cc:407] Connection header received
0806 09:21:36.807952 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:36.807982 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:36.808563 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:36.809036 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:36.811138 (+ 2102us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:36.812750 (+ 1612us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:36.815172 (+ 2422us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:36.816184 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:36.816427 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:36.822511 (+ 6084us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:36.822522 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:36.822543 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:36.822596 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:36.826954 (+ 4358us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:36.828014 (+ 1060us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:36.828022 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:36.828030 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:36.828239 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:36.828672 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:36.828677 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:36.828680 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:36.829411 (+ 731us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:36.829830 (+ 419us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:36.830320 (+ 490us) server_negotiation.cc:299] Negotiation successful
0806 09:21:36.830907 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":842,"thread_start_us":529,"threads_started":1}
I20260806 09:21:36.837154 21060 raft_consensus.cc:348] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c7b8cfc287438a84d4e08db673a4ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38007 } }
I20260806 09:21:36.837738 21060 raft_consensus.cc:374] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:36.837877 21060 raft_consensus.cc:729] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40c7b8cfc287438a84d4e08db673a4ab, State: Initialized, Role: FOLLOWER
I20260806 09:21:36.838603 21060 consensus_queue.cc:260] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c7b8cfc287438a84d4e08db673a4ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38007 } }
I20260806 09:21:36.838935 21060 raft_consensus.cc:388] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:36.839141 21060 raft_consensus.cc:482] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:36.839388 21060 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:36.842217 21060 raft_consensus.cc:504] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c7b8cfc287438a84d4e08db673a4ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38007 } }
I20260806 09:21:36.842891 21060 leader_election.cc:302] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 40c7b8cfc287438a84d4e08db673a4ab; no voters:
I20260806 09:21:36.843773 21060 leader_election.cc:288] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:36.844085 21066 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:36.848896 21066 raft_consensus.cc:686] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [term 1 LEADER]: Becoming Leader. State: Replica: 40c7b8cfc287438a84d4e08db673a4ab, State: Running, Role: LEADER
I20260806 09:21:36.849696 21060 sys_catalog.cc:564] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:36.849956 21066 consensus_queue.cc:237] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c7b8cfc287438a84d4e08db673a4ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38007 } }
I20260806 09:21:36.857383 21067 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "40c7b8cfc287438a84d4e08db673a4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c7b8cfc287438a84d4e08db673a4ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38007 } } }
I20260806 09:21:36.857800 21067 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:36.858388 21068 sys_catalog.cc:455] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 40c7b8cfc287438a84d4e08db673a4ab. Latest consensus state: current_term: 1 leader_uuid: "40c7b8cfc287438a84d4e08db673a4ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "40c7b8cfc287438a84d4e08db673a4ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38007 } } }
I20260806 09:21:36.858765 21068 sys_catalog.cc:458] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:36.858922 21083 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:36.866621 21083 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:36.879029 21083 catalog_manager.cc:1347] Generated new cluster ID: 0821624e7a344c16a912483dab7fb5b6
I20260806 09:21:36.879207 21083 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:36.917479 21083 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:36.919974 21083 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:36.934186 21083 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 40c7b8cfc287438a84d4e08db673a4ab: Generated new TSK 0
I20260806 09:21:36.935504 21083 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:21:37.008852 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38007
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:37.419840 21103 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:37.420192 21103 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:37.420343 21103 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:37.427860 21103 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:37.427990 21103 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:37.428181 21103 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:21:37.438519 21103 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:37.440152 21103 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:37.441612 21103 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:37.457777 21109 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:37.457726 21110 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:37.457756 21113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:37.458841 21103 server_base.cc:1034] running on GCE node
I20260806 09:21:37.459653 21103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:37.460918 21103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:37.462193 21103 hybrid_clock.cc:648] HybridClock initialized: now 1786008097462115 us; error 72 us; skew 500 ppm
I20260806 09:21:37.462558 21103 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:37 dist-test-slave-h3ft krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008097, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:37.466766 21103 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:21:37.468784 21103 webserver.cc:460] Webserver started at http://127.16.86.129:42891/ using document root <none> and password file <none>
I20260806 09:21:37.469615 21103 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:37.469789 21103 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:37.470146 21103 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:37.473430 21103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "bb1b76e74f8b4c9c9e5d03513b86c700"
format_stamp: "Formatted at 2026-08-06 09:21:37 on dist-test-slave-h3ft"
server_key: "f4b66b8c2d2e910095781a92744da6f5"
server_key_iv: "5510d2b13c9692998e9dca12ef0fe62e"
server_key_version: "encryptionkey@0"
I20260806 09:21:37.474341 21103 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "bb1b76e74f8b4c9c9e5d03513b86c700"
format_stamp: "Formatted at 2026-08-06 09:21:37 on dist-test-slave-h3ft"
server_key: "f4b66b8c2d2e910095781a92744da6f5"
server_key_iv: "5510d2b13c9692998e9dca12ef0fe62e"
server_key_version: "encryptionkey@0"
I20260806 09:21:37.481180 21103 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:21:37.485782 21120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:37.487802 21103 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:21:37.488050 21103 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "bb1b76e74f8b4c9c9e5d03513b86c700"
format_stamp: "Formatted at 2026-08-06 09:21:37 on dist-test-slave-h3ft"
server_key: "f4b66b8c2d2e910095781a92744da6f5"
server_key_iv: "5510d2b13c9692998e9dca12ef0fe62e"
server_key_version: "encryptionkey@0"
I20260806 09:21:37.488291 21103 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:37.531306 21103 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:37.536222 21103 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:37.536680 21103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:37.537848 21103 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:37.539554 21103 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:37.539673 21103 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:37.539840 21103 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:37.539918 21103 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:37.581295 21103 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:42077
I20260806 09:21:37.581421 21275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:42077 every 8 connection(s)
I20260806 09:21:37.582831 21103 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:21:37.593397 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 21103
I20260806 09:21:37.593653 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:21:37.594578 16730 external_mini_cluster.cc:1428] Setting key de9c41a60704bb2abf5230b85e678cdf
Aug 06 09:21:37 dist-test-slave-h3ft krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008097, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:37.609830 21278 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:37.586349 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:53247 (local address 127.16.86.190:38007)
0806 09:21:37.586941 (+ 592us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:37.586949 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:37.588280 (+ 1331us) server_negotiation.cc:407] Connection header received
0806 09:21:37.590292 (+ 2012us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:37.590306 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:37.590428 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:37.590565 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:37.592349 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:37.593759 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:37.596016 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:37.596575 (+ 559us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:37.596720 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:37.601124 (+ 4404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:37.601133 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:37.601141 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:37.601199 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:37.605460 (+ 4261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:37.606556 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:37.606564 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:37.606571 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:37.606758 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:37.607412 (+ 654us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:37.607419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:37.607424 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:37.607873 (+ 449us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:37.608048 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:37.609004 (+ 956us) server_negotiation.cc:299] Negotiation successful
0806 09:21:37.609319 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":428,"thread_start_us":278,"threads_started":1}
I20260806 09:21:37.610728 21279 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:37.587147 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:38007 (local address 127.16.86.129:53247)
0806 09:21:37.588047 (+ 900us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:37.588107 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:37.589943 (+ 1836us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:37.590815 (+ 872us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:37.590824 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:37.591496 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:37.592120 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:37.592139 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:37.594055 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:37.594059 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:37.595831 (+ 1772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:37.595853 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:37.596909 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:37.596917 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:37.597293 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:37.598211 (+ 918us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:37.598237 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:37.600809 (+ 2572us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:37.605711 (+ 4902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:37.605720 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:37.605738 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:37.606353 (+ 615us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:37.606932 (+ 579us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:37.606936 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:37.606939 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:37.607263 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:37.608051 (+ 788us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:37.608058 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:37.608507 (+ 449us) client_negotiation.cc:769] Sending connection context
0806 09:21:37.608804 (+ 297us) client_negotiation.cc:240] Negotiation successful
0806 09:21:37.609286 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":344,"threads_started":1}
I20260806 09:21:37.612871 21276 heartbeater.cc:347] Connected to a master server at 127.16.86.190:38007
I20260806 09:21:37.613400 21276 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:37.614473 21276 heartbeater.cc:511] Master 127.16.86.190:38007 requested a full tablet report, sending...
I20260806 09:21:37.617544 21014 ts_manager.cc:194] Registered new tserver with Master: bb1b76e74f8b4c9c9e5d03513b86c700 (127.16.86.129:42077)
I20260806 09:21:37.619526 21014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:53247
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:21:37.663410 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38007
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:38.098273 21282 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:38.098685 21282 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:38.098798 21282 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:38.107049 21282 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:38.107224 21282 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:38.107359 21282 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:21:38.118512 21282 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:38.120558 21282 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:38.122282 21282 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:38.139691 21290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:38.139605 21292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:38.139878 21294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:38.140832 21282 server_base.cc:1034] running on GCE node
I20260806 09:21:38.141673 21282 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:38.142987 21282 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:38.144309 21282 hybrid_clock.cc:648] HybridClock initialized: now 1786008098144249 us; error 43 us; skew 500 ppm
I20260806 09:21:38.144783 21282 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:38 dist-test-slave-h3ft krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008098, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:38.148983 21282 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:21:38.151242 21282 webserver.cc:460] Webserver started at http://127.16.86.130:44637/ using document root <none> and password file <none>
I20260806 09:21:38.152168 21282 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:38.152369 21282 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:38.152777 21282 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:38.156541 21282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "e694aa3c06e940c285f3146d47f98dfd"
format_stamp: "Formatted at 2026-08-06 09:21:38 on dist-test-slave-h3ft"
server_key: "468aa9254c2d7c7725c2cc4caf2fa82f"
server_key_iv: "25e4ace5d3acfe7734c8cf23eb60fd3f"
server_key_version: "encryptionkey@0"
I20260806 09:21:38.157506 21282 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "e694aa3c06e940c285f3146d47f98dfd"
format_stamp: "Formatted at 2026-08-06 09:21:38 on dist-test-slave-h3ft"
server_key: "468aa9254c2d7c7725c2cc4caf2fa82f"
server_key_iv: "25e4ace5d3acfe7734c8cf23eb60fd3f"
server_key_version: "encryptionkey@0"
I20260806 09:21:38.164287 21282 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:21:38.169348 21304 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:38.171173 21282 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:21:38.171372 21282 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "e694aa3c06e940c285f3146d47f98dfd"
format_stamp: "Formatted at 2026-08-06 09:21:38 on dist-test-slave-h3ft"
server_key: "468aa9254c2d7c7725c2cc4caf2fa82f"
server_key_iv: "25e4ace5d3acfe7734c8cf23eb60fd3f"
server_key_version: "encryptionkey@0"
I20260806 09:21:38.171623 21282 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:38.190810 21282 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:38.196262 21282 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:38.196578 21282 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:38.197773 21282 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:38.199579 21282 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:38.199702 21282 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:38.199874 21282 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:38.199955 21282 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:38.243238 21282 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:43665
I20260806 09:21:38.243315 21465 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:43665 every 8 connection(s)
I20260806 09:21:38.244966 21282 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:21:38.254653 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 21282
I20260806 09:21:38.254869 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:21:38.255512 16730 external_mini_cluster.cc:1428] Setting key 6ca0830f6607565d0fe8e66685058205
Aug 06 09:21:38 dist-test-slave-h3ft krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008098, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:38.273141 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:38.248611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:60447 (local address 127.16.86.190:38007)
0806 09:21:38.249224 (+ 613us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:38.249229 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:38.250791 (+ 1562us) server_negotiation.cc:407] Connection header received
0806 09:21:38.253097 (+ 2306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:38.253103 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:38.253221 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:38.253387 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:38.255984 (+ 2597us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:38.257026 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:38.259480 (+ 2454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:38.260131 (+ 651us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:38.260374 (+ 243us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:38.265928 (+ 5554us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:38.265934 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:38.265937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:38.265980 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:38.269236 (+ 3256us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:38.270270 (+ 1034us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:38.270280 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:38.270283 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:38.270430 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:38.270963 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:38.270968 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:38.270970 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:38.271372 (+ 402us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:38.271555 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:38.272418 (+ 863us) server_negotiation.cc:299] Negotiation successful
0806 09:21:38.272700 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":306,"threads_started":1}
I20260806 09:21:38.274433 21469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:38.249397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:38007 (local address 127.16.86.130:60447)
0806 09:21:38.250521 (+ 1124us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:38.250585 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:38.252721 (+ 2136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:38.253745 (+ 1024us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:38.253754 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:38.254620 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:38.255628 (+ 1008us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:38.255662 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:38.257239 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:38.257244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:38.259245 (+ 2001us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:38.259267 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:38.260494 (+ 1227us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:38.260506 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:38.260923 (+ 417us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:38.262025 (+ 1102us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:38.262065 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:38.265581 (+ 3516us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:38.269440 (+ 3859us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:38.269453 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:38.269482 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:38.270115 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:38.270566 (+ 451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:38.270572 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:38.270575 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:38.270810 (+ 235us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:38.271525 (+ 715us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:38.271535 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:38.271996 (+ 461us) client_negotiation.cc:769] Sending connection context
0806 09:21:38.272316 (+ 320us) client_negotiation.cc:240] Negotiation successful
0806 09:21:38.273027 (+ 711us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":530,"threads_started":1}
I20260806 09:21:38.276467 21466 heartbeater.cc:347] Connected to a master server at 127.16.86.190:38007
I20260806 09:21:38.276865 21466 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:38.278031 21466 heartbeater.cc:511] Master 127.16.86.190:38007 requested a full tablet report, sending...
I20260806 09:21:38.280067 21014 ts_manager.cc:194] Registered new tserver with Master: e694aa3c06e940c285f3146d47f98dfd (127.16.86.130:43665)
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
I20260806 09:21:38.281136 21014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:60447
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:21:38.315973 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38007
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:41537
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:21:38.623854 21276 heartbeater.cc:503] Master 127.16.86.190:38007 was elected leader, sending a full tablet report...
W20260806 09:21:38.735344 21472 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:38.735754 21472 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:38.735947 21472 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:38.745091 21472 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:38.745268 21472 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:38.745491 21472 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:21:38.758353 21472 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:41537
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38007
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:38.760403 21472 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:38.762221 21472 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:38.780378 21478 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:38.780587 21479 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:38.780563 21482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:38.782090 21472 server_base.cc:1034] running on GCE node
I20260806 09:21:38.783129 21472 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:38.784526 21472 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:38.785845 21472 hybrid_clock.cc:648] HybridClock initialized: now 1786008098785751 us; error 82 us; skew 500 ppm
I20260806 09:21:38.786499 21472 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:38 dist-test-slave-h3ft krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008098, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:38.791320 21472 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:21:38.793946 21472 webserver.cc:460] Webserver started at http://127.16.86.131:34679/ using document root <none> and password file <none>
I20260806 09:21:38.794970 21472 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:38.795254 21472 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:38.795675 21472 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:38.799921 21472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "51eff70c2da2470da9b5fd5f0c75b9b5"
format_stamp: "Formatted at 2026-08-06 09:21:38 on dist-test-slave-h3ft"
server_key: "fd893e536dadbed81b8d69011d16af3d"
server_key_iv: "0e1887cce7e61bca752de3f7eb106f68"
server_key_version: "encryptionkey@0"
I20260806 09:21:38.800990 21472 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "51eff70c2da2470da9b5fd5f0c75b9b5"
format_stamp: "Formatted at 2026-08-06 09:21:38 on dist-test-slave-h3ft"
server_key: "fd893e536dadbed81b8d69011d16af3d"
server_key_iv: "0e1887cce7e61bca752de3f7eb106f68"
server_key_version: "encryptionkey@0"
I20260806 09:21:38.808260 21472 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260806 09:21:38.813205 21494 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:38.815294 21472 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20260806 09:21:38.815554 21472 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "51eff70c2da2470da9b5fd5f0c75b9b5"
format_stamp: "Formatted at 2026-08-06 09:21:38 on dist-test-slave-h3ft"
server_key: "fd893e536dadbed81b8d69011d16af3d"
server_key_iv: "0e1887cce7e61bca752de3f7eb106f68"
server_key_version: "encryptionkey@0"
I20260806 09:21:38.815819 21472 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:38.836076 21472 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:38.842722 21472 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:38.843354 21472 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:38.845410 21472 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:38.848278 21472 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:38.848417 21472 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:38.848634 21472 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:38.848727 21472 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:38.898339 21472 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:39119
I20260806 09:21:38.898391 21648 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:39119 every 8 connection(s)
I20260806 09:21:38.900276 21472 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:21:38.909555 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 21472
I20260806 09:21:38.909777 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:21:38.910358 16730 external_mini_cluster.cc:1428] Setting key d7a31479478794f231a7432b373c8517
Aug 06 09:21:38 dist-test-slave-h3ft krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008098, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:38.926666 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:38.903903 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:57267 (local address 127.16.86.190:38007)
0806 09:21:38.904556 (+ 653us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:38.904562 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:38.906005 (+ 1443us) server_negotiation.cc:407] Connection header received
0806 09:21:38.908206 (+ 2201us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:38.908211 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:38.908328 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:38.908459 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:38.910783 (+ 2324us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:38.912093 (+ 1310us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:38.914192 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:38.914781 (+ 589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:38.914919 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:38.918797 (+ 3878us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:38.918804 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:38.918808 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:38.918901 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:38.922604 (+ 3703us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:38.923548 (+ 944us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:38.923553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:38.923555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:38.923755 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:38.924304 (+ 549us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:38.924309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:38.924311 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:38.924691 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:38.924894 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:38.925914 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0806 09:21:38.926228 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":366,"threads_started":1}
I20260806 09:21:38.927867 21653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:38.904691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:38007 (local address 127.16.86.131:57267)
0806 09:21:38.905795 (+ 1104us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:38.905848 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:38.907882 (+ 2034us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:38.908737 (+ 855us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:38.908746 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:38.909569 (+ 823us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:38.910483 (+ 914us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:38.910514 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:38.912261 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:38.912267 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:38.914016 (+ 1749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:38.914030 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:38.915086 (+ 1056us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:38.915094 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:38.915333 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:38.916172 (+ 839us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:38.916195 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:38.918513 (+ 2318us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:38.922779 (+ 4266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:38.922787 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:38.922801 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:38.923384 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:38.923905 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:38.923908 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:38.923911 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:38.924101 (+ 190us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:38.924847 (+ 746us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:38.924855 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:38.925338 (+ 483us) client_negotiation.cc:769] Sending connection context
0806 09:21:38.925674 (+ 336us) client_negotiation.cc:240] Negotiation successful
0806 09:21:38.926265 (+ 591us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":756,"thread_start_us":440,"threads_started":1}
I20260806 09:21:38.929827 21649 heartbeater.cc:347] Connected to a master server at 127.16.86.190:38007
I20260806 09:21:38.930171 21649 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:38.931145 21649 heartbeater.cc:511] Master 127.16.86.190:38007 requested a full tablet report, sending...
I20260806 09:21:38.932917 21014 ts_manager.cc:194] Registered new tserver with Master: 51eff70c2da2470da9b5fd5f0c75b9b5 (127.16.86.131:39119)
I20260806 09:21:38.933815 21014 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:57267
I20260806 09:21:38.941952 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:21:38 dist-test-slave-h3ft krb5kdc[20955](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008098, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Aug 06 09:21:39 dist-test-slave-h3ft krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008098, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:39.016172 21652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:38.998316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46748 (local address 127.16.86.190:38007)
0806 09:21:38.998662 (+ 346us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:38.998672 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:38.999233 (+ 561us) server_negotiation.cc:407] Connection header received
0806 09:21:38.999457 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:38.999463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:38.999577 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:38.999707 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:39.001393 (+ 1686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:39.002279 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:39.003826 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:39.004376 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:39.004513 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:39.009338 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:39.009344 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:39.009348 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:39.009401 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:39.012677 (+ 3276us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:39.013577 (+ 900us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:39.013582 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:39.013585 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:39.013782 (+ 197us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:39.014313 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:39.014317 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:39.014320 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:39.014762 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:39.015090 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:39.015492 (+ 402us) server_negotiation.cc:299] Negotiation successful
0806 09:21:39.015722 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":107}
W20260806 09:21:39.017199 21014 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:46748
I20260806 09:21:39.019913 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 21103
Aug 06 09:21:39 dist-test-slave-h3ft krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008098, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:21:39.040343 21663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:39.021969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49918 (local address 127.16.86.129:42077)
0806 09:21:39.022696 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:39.022704 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:39.022734 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:21:39.022801 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:39.022807 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:39.023098 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:39.023261 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:39.024762 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:39.025822 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:39.027397 (+ 1575us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:39.028018 (+ 621us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:39.028156 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:39.033144 (+ 4988us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:39.033166 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:39.033198 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:39.033259 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:39.036810 (+ 3551us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:39.037773 (+ 963us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:39.037783 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:39.037785 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:39.038010 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:39.038429 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:39.038433 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:39.038435 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:39.038882 (+ 447us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:39.039320 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:39.039689 (+ 369us) server_negotiation.cc:299] Negotiation successful
0806 09:21:39.039994 (+ 305us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":554,"thread_start_us":367,"threads_started":1}
W20260806 09:21:39.042218 21266 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:49918
W20260806 09:21:39.043931 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 21103: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:21:39.044260 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 21103
I20260806 09:21:39.070719 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 21282
Aug 06 09:21:39 dist-test-slave-h3ft krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008098, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:21:39.092722 21664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:39.072221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38878 (local address 127.16.86.130:43665)
0806 09:21:39.072984 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:39.072994 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:39.073041 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:21:39.073131 (+ 90us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:39.073139 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:39.073441 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:39.073624 (+ 183us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:39.075182 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:39.076505 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:39.078029 (+ 1524us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:39.078808 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:39.078955 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:39.084978 (+ 6023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:39.085002 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:39.085025 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:39.085085 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:39.089083 (+ 3998us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:39.090076 (+ 993us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:39.090085 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:39.090088 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:39.090271 (+ 183us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:39.090665 (+ 394us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:39.090668 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:39.090670 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:39.091248 (+ 578us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:39.091544 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:39.091885 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 09:21:39.092266 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":353,"threads_started":1}
W20260806 09:21:39.094471 21459 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:38878
W20260806 09:21:39.096053 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 21282: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:21:39.096297 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 21282
I20260806 09:21:39.124275 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 21472
Aug 06 09:21:39 dist-test-slave-h3ft krb5kdc[20955](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008098, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:21:39.144745 21665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:39.125633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52838 (local address 127.16.86.131:39119)
0806 09:21:39.126508 (+ 875us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:39.126516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:39.126557 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:21:39.126641 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:39.126649 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:39.126938 (+ 289us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:39.127180 (+ 242us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:39.128756 (+ 1576us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:39.129992 (+ 1236us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:39.131461 (+ 1469us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:39.132199 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:39.132343 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:39.137340 (+ 4997us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:39.137357 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:39.137378 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:39.137432 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:39.141172 (+ 3740us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:39.142188 (+ 1016us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:39.142199 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:39.142202 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:39.142402 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:39.142930 (+ 528us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:39.142934 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:39.142937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:39.143473 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:39.143836 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:39.144111 (+ 275us) server_negotiation.cc:299] Negotiation successful
0806 09:21:39.144379 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":674,"thread_start_us":462,"threads_started":1}
W20260806 09:21:39.146327 21642 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:52838
W20260806 09:21:39.148106 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 21472: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:21:39.148346 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 21472
I20260806 09:21:39.174317 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 20975
I20260806 09:21:39.176007 21054 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:39.285004 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 20975
2026-08-06T09:21:39Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4691 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:21:39 dist-test-slave-h3ft krb5kdc[21672](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:21:39 dist-test-slave-h3ft krb5kdc[21672](info): set up 2 sockets
Aug 06 09:21:39 dist-test-slave-h3ft krb5kdc[21672](info): commencing operation
krb5kdc: starting...
W20260806 09:21:40.635541 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.295s user 0.000s sys 0.033s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:21:40 dist-test-slave-h3ft krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:21:40Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:21:40.928531 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:38979
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:38979
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:41.303272 21693 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:41.303673 21693 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:41.303846 21693 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:41.311348 21693 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:41.311507 21693 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:41.311626 21693 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:41.311728 21693 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:41.311820 21693 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:21:41.322134 21693 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:38979
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:38979
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:41.323650 21693 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:41.325115 21693 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:41.338182 21703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:41.338305 21708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:41.338351 21704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:41.339865 21693 server_base.cc:1034] running on GCE node
I20260806 09:21:41.340999 21693 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:41.342897 21693 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:41.344182 21693 hybrid_clock.cc:648] HybridClock initialized: now 1786008101344101 us; error 62 us; skew 500 ppm
I20260806 09:21:41.344623 21693 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:41 dist-test-slave-h3ft krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008101, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:41.349146 21693 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:21:41.351615 21693 webserver.cc:460] Webserver started at http://127.16.86.190:44571/ using document root <none> and password file <none>
I20260806 09:21:41.352752 21693 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:41.352957 21693 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:41.353456 21693 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:41.357914 21693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "31aa1c71c1424e0cbfa1b5c5936fee37"
format_stamp: "Formatted at 2026-08-06 09:21:41 on dist-test-slave-h3ft"
server_key: "19fe7095be44cf221e580cae6b41e644"
server_key_iv: "26b71d6e25171b4b7317d0e402f422b5"
server_key_version: "encryptionkey@0"
I20260806 09:21:41.359308 21693 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "31aa1c71c1424e0cbfa1b5c5936fee37"
format_stamp: "Formatted at 2026-08-06 09:21:41 on dist-test-slave-h3ft"
server_key: "19fe7095be44cf221e580cae6b41e644"
server_key_iv: "26b71d6e25171b4b7317d0e402f422b5"
server_key_version: "encryptionkey@0"
I20260806 09:21:41.367594 21693 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260806 09:21:41.372926 21717 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:41.374848 21693 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:21:41.375152 21693 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "31aa1c71c1424e0cbfa1b5c5936fee37"
format_stamp: "Formatted at 2026-08-06 09:21:41 on dist-test-slave-h3ft"
server_key: "19fe7095be44cf221e580cae6b41e644"
server_key_iv: "26b71d6e25171b4b7317d0e402f422b5"
server_key_version: "encryptionkey@0"
I20260806 09:21:41.375443 21693 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:41.397974 21693 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:41.402935 21693 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:41.403307 21693 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:41.425206 21693 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:38979
I20260806 09:21:41.425267 21788 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:38979 every 8 connection(s)
I20260806 09:21:41.426766 21693 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:41.430107 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 21693
I20260806 09:21:41.430387 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:41.431119 16730 external_mini_cluster.cc:1428] Setting key 33d45abf946ee50834722684416bcc6e
I20260806 09:21:41.431947 21790 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:21:41 dist-test-slave-h3ft krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008100, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:41.449926 21790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37: Bootstrap starting.
I20260806 09:21:41.454377 21790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:41.455794 21790 log.cc:824] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:41.458312 21795 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:41.433379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52234 (local address 127.16.86.190:38979)
0806 09:21:41.434269 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:41.434282 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:41.434334 (+ 52us) server_negotiation.cc:407] Connection header received
0806 09:21:41.436182 (+ 1848us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:41.436208 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:41.436699 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:41.437087 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:41.439592 (+ 2505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:41.441126 (+ 1534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:41.443130 (+ 2004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:41.443912 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:41.444063 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:41.449703 (+ 5640us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:41.449713 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:41.449733 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:41.449786 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:41.453586 (+ 3800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:41.454436 (+ 850us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:41.454444 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:41.454451 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:41.454711 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:41.455191 (+ 480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:41.455196 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:41.455200 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:41.455852 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:41.456216 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:41.456664 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 09:21:41.457185 (+ 521us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":648,"thread_start_us":409,"threads_started":1}
I20260806 09:21:41.459861 21790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37: No bootstrap required, opened a new log
I20260806 09:21:41.469321 21790 raft_consensus.cc:348] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31aa1c71c1424e0cbfa1b5c5936fee37" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38979 } }
I20260806 09:21:41.469828 21790 raft_consensus.cc:374] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:41.469923 21790 raft_consensus.cc:729] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31aa1c71c1424e0cbfa1b5c5936fee37, State: Initialized, Role: FOLLOWER
I20260806 09:21:41.470647 21790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31aa1c71c1424e0cbfa1b5c5936fee37" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38979 } }
I20260806 09:21:41.470912 21790 raft_consensus.cc:388] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:41.471101 21790 raft_consensus.cc:482] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:41.471271 21790 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:41.474117 21790 raft_consensus.cc:504] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31aa1c71c1424e0cbfa1b5c5936fee37" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38979 } }
I20260806 09:21:41.474927 21790 leader_election.cc:302] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 31aa1c71c1424e0cbfa1b5c5936fee37; no voters:
I20260806 09:21:41.475798 21790 leader_election.cc:288] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:41.476047 21797 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:41.476976 21797 raft_consensus.cc:686] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [term 1 LEADER]: Becoming Leader. State: Replica: 31aa1c71c1424e0cbfa1b5c5936fee37, State: Running, Role: LEADER
I20260806 09:21:41.477675 21797 consensus_queue.cc:237] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31aa1c71c1424e0cbfa1b5c5936fee37" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38979 } }
I20260806 09:21:41.478499 21790 sys_catalog.cc:564] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:41.481252 21798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "31aa1c71c1424e0cbfa1b5c5936fee37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31aa1c71c1424e0cbfa1b5c5936fee37" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38979 } } }
I20260806 09:21:41.481315 21799 sys_catalog.cc:455] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 31aa1c71c1424e0cbfa1b5c5936fee37. Latest consensus state: current_term: 1 leader_uuid: "31aa1c71c1424e0cbfa1b5c5936fee37" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "31aa1c71c1424e0cbfa1b5c5936fee37" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38979 } } }
I20260806 09:21:41.482288 21799 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:41.482295 21798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:41.484381 21803 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:41.492769 21803 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:41.503914 21803 catalog_manager.cc:1347] Generated new cluster ID: 7d355d0b1ae74875a2b4da04d3a1cdb3
I20260806 09:21:41.504070 21803 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:41.527369 21803 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:41.529426 21803 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:41.542793 21803 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 31aa1c71c1424e0cbfa1b5c5936fee37: Generated new TSK 0
I20260806 09:21:41.544020 21803 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:21:41.628183 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38979
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:42.064644 21822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:42.065110 21822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:42.065294 21822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:42.073805 21822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:42.074028 21822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:42.074244 21822 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:21:42.087044 21822 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:42.089172 21822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:42.091264 21822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:42.109685 21831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:42.109750 21829 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:42.109675 21828 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:42.111124 21822 server_base.cc:1034] running on GCE node
I20260806 09:21:42.112008 21822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:42.113303 21822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:42.114578 21822 hybrid_clock.cc:648] HybridClock initialized: now 1786008102114506 us; error 51 us; skew 500 ppm
I20260806 09:21:42.115165 21822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:42 dist-test-slave-h3ft krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008102, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:42.123785 21822 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:21:42.125993 21822 webserver.cc:460] Webserver started at http://127.16.86.129:42925/ using document root <none> and password file <none>
I20260806 09:21:42.127076 21822 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:42.127247 21822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:42.127681 21822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:42.131747 21822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "b1fcf51eff04469da6f70a3646fdee04"
format_stamp: "Formatted at 2026-08-06 09:21:42 on dist-test-slave-h3ft"
server_key: "937ed895f01d24ce1150848f2af17b68"
server_key_iv: "c7db2fbcfde00d79fbb55d10a38842dd"
server_key_version: "encryptionkey@0"
I20260806 09:21:42.132894 21822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "b1fcf51eff04469da6f70a3646fdee04"
format_stamp: "Formatted at 2026-08-06 09:21:42 on dist-test-slave-h3ft"
server_key: "937ed895f01d24ce1150848f2af17b68"
server_key_iv: "c7db2fbcfde00d79fbb55d10a38842dd"
server_key_version: "encryptionkey@0"
I20260806 09:21:42.140152 21822 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260806 09:21:42.145407 21840 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:42.147867 21822 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:21:42.148241 21822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "b1fcf51eff04469da6f70a3646fdee04"
format_stamp: "Formatted at 2026-08-06 09:21:42 on dist-test-slave-h3ft"
server_key: "937ed895f01d24ce1150848f2af17b68"
server_key_iv: "c7db2fbcfde00d79fbb55d10a38842dd"
server_key_version: "encryptionkey@0"
I20260806 09:21:42.148561 21822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:42.168192 21822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:42.174192 21822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:42.174615 21822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:42.176048 21822 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:42.177917 21822 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:42.178012 21822 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:42.178201 21822 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:42.178277 21822 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:42.222535 21822 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:42201
I20260806 09:21:42.222611 21998 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:42201 every 8 connection(s)
I20260806 09:21:42.224185 21822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:21:42.233520 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 21822
I20260806 09:21:42.233804 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:21:42.234504 16730 external_mini_cluster.cc:1428] Setting key b954f2bfda370ee43b7aaea500db5142
Aug 06 09:21:42 dist-test-slave-h3ft krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008102, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:42.252840 22001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:42.227925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:39705 (local address 127.16.86.190:38979)
0806 09:21:42.229027 (+ 1102us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:42.229037 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:42.230264 (+ 1227us) server_negotiation.cc:407] Connection header received
0806 09:21:42.232530 (+ 2266us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:42.232537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:42.232669 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:42.232811 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:42.235029 (+ 2218us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:42.236038 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:42.238025 (+ 1987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:42.238498 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:42.238653 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:42.244634 (+ 5981us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:42.244645 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:42.244653 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:42.244716 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:42.248432 (+ 3716us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:42.249793 (+ 1361us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:42.249798 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:42.249801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:42.249972 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:42.250601 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:42.250605 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:42.250607 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:42.250930 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:42.251174 (+ 244us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:42.252242 (+ 1068us) server_negotiation.cc:299] Negotiation successful
0806 09:21:42.252490 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":553,"threads_started":1}
I20260806 09:21:42.254971 22004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:42.228840 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:38979 (local address 127.16.86.129:39705)
0806 09:21:42.230011 (+ 1171us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:42.230073 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:42.232153 (+ 2080us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:42.233140 (+ 987us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:42.233150 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:42.233927 (+ 777us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:42.234793 (+ 866us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:42.234816 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:42.236237 (+ 1421us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:42.236241 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:42.237839 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:42.237853 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:42.238790 (+ 937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:42.238799 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:42.239225 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:42.240116 (+ 891us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:42.240144 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:42.244179 (+ 4035us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:42.248708 (+ 4529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:42.248724 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:42.248748 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:42.249607 (+ 859us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:42.250122 (+ 515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:42.250129 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:42.250135 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:42.250445 (+ 310us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:42.251176 (+ 731us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:42.251188 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:42.251774 (+ 586us) client_negotiation.cc:769] Sending connection context
0806 09:21:42.252143 (+ 369us) client_negotiation.cc:240] Negotiation successful
0806 09:21:42.253112 (+ 969us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":511,"threads_started":1}
I20260806 09:21:42.257483 21999 heartbeater.cc:347] Connected to a master server at 127.16.86.190:38979
I20260806 09:21:42.257925 21999 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:42.258971 21999 heartbeater.cc:511] Master 127.16.86.190:38979 requested a full tablet report, sending...
I20260806 09:21:42.262593 21739 ts_manager.cc:194] Registered new tserver with Master: b1fcf51eff04469da6f70a3646fdee04 (127.16.86.129:42201)
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
I20260806 09:21:42.265128 21739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:39705
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:21:42.304070 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38979
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:42.679399 22008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:42.679921 22008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:42.680146 22008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:42.688915 22008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:42.689077 22008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:42.689286 22008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:21:42.700866 22008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:42.702700 22008 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:42.704653 22008 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:42.721480 22013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:42.721604 22014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:42.721536 22017 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:42.725061 22008 server_base.cc:1034] running on GCE node
I20260806 09:21:42.726089 22008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:42.727761 22008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:42.729034 22008 hybrid_clock.cc:648] HybridClock initialized: now 1786008102728960 us; error 57 us; skew 500 ppm
I20260806 09:21:42.729509 22008 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:42 dist-test-slave-h3ft krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008102, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:42.733980 22008 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:21:42.736347 22008 webserver.cc:460] Webserver started at http://127.16.86.130:38117/ using document root <none> and password file <none>
I20260806 09:21:42.737383 22008 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:42.737560 22008 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:42.737989 22008 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:42.741923 22008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "7df4247084834bef94a5b7b0890b246b"
format_stamp: "Formatted at 2026-08-06 09:21:42 on dist-test-slave-h3ft"
server_key: "3f1982c4d152b492886962d98a1b66ce"
server_key_iv: "6229f2d7979b4fdcbc6497b1e0a7d2f8"
server_key_version: "encryptionkey@0"
I20260806 09:21:42.742898 22008 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "7df4247084834bef94a5b7b0890b246b"
format_stamp: "Formatted at 2026-08-06 09:21:42 on dist-test-slave-h3ft"
server_key: "3f1982c4d152b492886962d98a1b66ce"
server_key_iv: "6229f2d7979b4fdcbc6497b1e0a7d2f8"
server_key_version: "encryptionkey@0"
I20260806 09:21:42.750078 22008 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 09:21:42.755252 22028 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:42.757054 22008 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:21:42.757233 22008 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "7df4247084834bef94a5b7b0890b246b"
format_stamp: "Formatted at 2026-08-06 09:21:42 on dist-test-slave-h3ft"
server_key: "3f1982c4d152b492886962d98a1b66ce"
server_key_iv: "6229f2d7979b4fdcbc6497b1e0a7d2f8"
server_key_version: "encryptionkey@0"
I20260806 09:21:42.757457 22008 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:42.772935 22008 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:42.777427 22008 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:42.777709 22008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:42.778860 22008 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:42.780606 22008 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:42.780694 22008 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:42.780783 22008 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:42.780838 22008 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:42.821899 22008 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:33635
I20260806 09:21:42.822008 22187 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:33635 every 8 connection(s)
I20260806 09:21:42.823403 22008 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:21:42.832944 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 22008
I20260806 09:21:42.833237 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:21:42.833884 16730 external_mini_cluster.cc:1428] Setting key 1533a8eefb789eb8a24348f3a0314ce4
Aug 06 09:21:42 dist-test-slave-h3ft krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008102, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:42.852836 22192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:42.827775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:38979 (local address 127.16.86.130:56325)
0806 09:21:42.828696 (+ 921us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:42.828754 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:42.830569 (+ 1815us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:42.831495 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:42.831504 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:42.832089 (+ 585us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:42.832728 (+ 639us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:42.832748 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:42.834410 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:42.834415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:42.836726 (+ 2311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:42.836745 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:42.837869 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:42.837875 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:42.838101 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:42.838951 (+ 850us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:42.838975 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:42.841344 (+ 2369us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:42.846349 (+ 5005us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:42.846365 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:42.846388 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:42.847206 (+ 818us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:42.848010 (+ 804us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:42.848019 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:42.848025 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:42.848435 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:42.849294 (+ 859us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:42.849306 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:42.850037 (+ 731us) client_negotiation.cc:769] Sending connection context
0806 09:21:42.850500 (+ 463us) client_negotiation.cc:240] Negotiation successful
0806 09:21:42.851293 (+ 793us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":364,"threads_started":1}
I20260806 09:21:42.854521 22191 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:42.826945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:56325 (local address 127.16.86.190:38979)
0806 09:21:42.827600 (+ 655us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:42.827608 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:42.828922 (+ 1314us) server_negotiation.cc:407] Connection header received
0806 09:21:42.830900 (+ 1978us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:42.830906 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:42.831073 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:42.831229 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:42.832975 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:42.834178 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:42.836913 (+ 2735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:42.837594 (+ 681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:42.837756 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:42.841640 (+ 3884us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:42.841651 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:42.841660 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:42.841715 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:42.846074 (+ 4359us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:42.847411 (+ 1337us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:42.847420 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:42.847426 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:42.847771 (+ 345us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:42.848602 (+ 831us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:42.848608 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:42.848613 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:42.849149 (+ 536us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:42.849352 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:42.853869 (+ 4517us) server_negotiation.cc:299] Negotiation successful
0806 09:21:42.854193 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":487,"thread_start_us":333,"threads_started":1}
I20260806 09:21:42.856362 22188 heartbeater.cc:347] Connected to a master server at 127.16.86.190:38979
I20260806 09:21:42.856843 22188 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:42.857885 22188 heartbeater.cc:511] Master 127.16.86.190:38979 requested a full tablet report, sending...
I20260806 09:21:42.860201 21739 ts_manager.cc:194] Registered new tserver with Master: 7df4247084834bef94a5b7b0890b246b (127.16.86.130:33635)
I20260806 09:21:42.861655 21739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:56325
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:21:42.899987 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38979
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:40247
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:43.255985 22200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:43.256294 22200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:43.256392 22200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:43.263785 22200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:43.263912 22200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:43.264034 22200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:21:43.269691 21999 heartbeater.cc:503] Master 127.16.86.190:38979 was elected leader, sending a full tablet report...
I20260806 09:21:43.274923 22200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:40247
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38979
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:43.276424 22200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:43.277822 22200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:43.293627 22205 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:43.293740 22208 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:43.293763 22206 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:43.295264 22200 server_base.cc:1034] running on GCE node
I20260806 09:21:43.296181 22200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:43.297448 22200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:43.298751 22200 hybrid_clock.cc:648] HybridClock initialized: now 1786008103298687 us; error 50 us; skew 500 ppm
I20260806 09:21:43.299311 22200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:43 dist-test-slave-h3ft krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008103, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:43.303653 22200 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:21:43.305918 22200 webserver.cc:460] Webserver started at http://127.16.86.131:46243/ using document root <none> and password file <none>
I20260806 09:21:43.306903 22200 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:43.307147 22200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:43.307569 22200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:43.311585 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "3f480482f24d4b9c9232a52dd317f8e6"
format_stamp: "Formatted at 2026-08-06 09:21:43 on dist-test-slave-h3ft"
server_key: "5f603b4cf1ad46b0672442cdda1b6354"
server_key_iv: "495263dd753edde9b85a65e8a74a805b"
server_key_version: "encryptionkey@0"
I20260806 09:21:43.312722 22200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "3f480482f24d4b9c9232a52dd317f8e6"
format_stamp: "Formatted at 2026-08-06 09:21:43 on dist-test-slave-h3ft"
server_key: "5f603b4cf1ad46b0672442cdda1b6354"
server_key_iv: "495263dd753edde9b85a65e8a74a805b"
server_key_version: "encryptionkey@0"
I20260806 09:21:43.319868 22200 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260806 09:21:43.324729 22221 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:43.326961 22200 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:21:43.327332 22200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "3f480482f24d4b9c9232a52dd317f8e6"
format_stamp: "Formatted at 2026-08-06 09:21:43 on dist-test-slave-h3ft"
server_key: "5f603b4cf1ad46b0672442cdda1b6354"
server_key_iv: "495263dd753edde9b85a65e8a74a805b"
server_key_version: "encryptionkey@0"
I20260806 09:21:43.327677 22200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:43.361057 22200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:43.367118 22200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:43.367543 22200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:43.368904 22200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:43.370826 22200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:43.370970 22200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:43.371222 22200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:43.371320 22200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:43.416682 22200 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:37145
I20260806 09:21:43.416801 22381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:37145 every 8 connection(s)
I20260806 09:21:43.418525 22200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:21:43.418900 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 22200
I20260806 09:21:43.419179 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:21:43.419947 16730 external_mini_cluster.cc:1428] Setting key 754a1166db876c9a4d0e68e7f031497e
Aug 06 09:21:43 dist-test-slave-h3ft krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008103, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:43.444866 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:43.422051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:52117 (local address 127.16.86.190:38979)
0806 09:21:43.422681 (+ 630us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:43.422689 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:43.424095 (+ 1406us) server_negotiation.cc:407] Connection header received
0806 09:21:43.426315 (+ 2220us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:43.426321 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:43.426452 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:43.426583 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:43.428549 (+ 1966us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.429509 (+ 960us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.431803 (+ 2294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.432283 (+ 480us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.432447 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:43.437519 (+ 5072us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:43.437526 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:43.437533 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:43.437586 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:43.440707 (+ 3121us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:43.441853 (+ 1146us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:43.441857 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:43.441860 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:43.442009 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:43.442538 (+ 529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:43.442541 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:43.442543 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:43.442871 (+ 328us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:43.443086 (+ 215us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:43.444239 (+ 1153us) server_negotiation.cc:299] Negotiation successful
0806 09:21:43.444468 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":483,"thread_start_us":309,"threads_started":1}
I20260806 09:21:43.446256 22385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:43.422765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:38979 (local address 127.16.86.131:52117)
0806 09:21:43.423870 (+ 1105us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:43.423953 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:43.425995 (+ 2042us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:43.426903 (+ 908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:43.426913 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:43.427630 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:43.428350 (+ 720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:43.428369 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.429677 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.429685 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:43.431518 (+ 1833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:43.431550 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.432592 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.432602 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:43.432925 (+ 323us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:43.433875 (+ 950us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:43.433901 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:43.437166 (+ 3265us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:43.440972 (+ 3806us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:43.440983 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:43.441008 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:43.441677 (+ 669us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:43.442142 (+ 465us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:43.442146 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:43.442149 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:43.442411 (+ 262us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:43.443168 (+ 757us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:43.443176 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:43.443700 (+ 524us) client_negotiation.cc:769] Sending connection context
0806 09:21:43.444062 (+ 362us) client_negotiation.cc:240] Negotiation successful
0806 09:21:43.444667 (+ 605us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":425,"threads_started":1}
I20260806 09:21:43.448724 22382 heartbeater.cc:347] Connected to a master server at 127.16.86.190:38979
I20260806 09:21:43.449122 22382 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:43.450131 22382 heartbeater.cc:511] Master 127.16.86.190:38979 requested a full tablet report, sending...
I20260806 09:21:43.452091 21739 ts_manager.cc:194] Registered new tserver with Master: 3f480482f24d4b9c9232a52dd317f8e6 (127.16.86.131:37145)
I20260806 09:21:43.453339 21739 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:52117
I20260806 09:21:43.464228 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:21:43 dist-test-slave-h3ft krb5kdc[21672](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:21:43 dist-test-slave-h3ft krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:21:43.539505 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:43.518726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43770 (local address 127.16.86.129:42201)
0806 09:21:43.519676 (+ 950us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:43.519685 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:43.519717 (+ 32us) server_negotiation.cc:407] Connection header received
0806 09:21:43.519798 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:43.519807 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:43.520166 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:43.520391 (+ 225us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:43.522358 (+ 1967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.523800 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.525294 (+ 1494us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.526031 (+ 737us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.526206 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:43.531432 (+ 5226us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:43.531452 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:43.531480 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:43.531534 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:43.535594 (+ 4060us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:43.536462 (+ 868us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:43.536471 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:43.536473 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:43.536622 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:43.537055 (+ 433us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:43.537062 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:43.537066 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:43.537662 (+ 596us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:43.538021 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:43.538614 (+ 593us) server_negotiation.cc:299] Negotiation successful
0806 09:21:43.538950 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":711,"thread_start_us":473,"threads_started":1}
Aug 06 09:21:43 dist-test-slave-h3ft krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:43.563848 22384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:43.546215 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52246 (local address 127.16.86.190:38979)
0806 09:21:43.546474 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:43.546484 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:43.546890 (+ 406us) server_negotiation.cc:407] Connection header received
0806 09:21:43.547232 (+ 342us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:43.547237 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:43.547336 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:43.547459 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:43.548990 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.549817 (+ 827us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.551276 (+ 1459us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.551714 (+ 438us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.551865 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:43.556847 (+ 4982us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:43.556853 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:43.556856 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:43.556906 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:43.560365 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:43.561258 (+ 893us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:43.561270 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:43.561273 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:43.561437 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:43.561910 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:43.561921 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:43.561923 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:43.562369 (+ 446us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:43.562698 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:43.563123 (+ 425us) server_negotiation.cc:299] Negotiation successful
0806 09:21:43.563437 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 09:21:43.564677 21739 server_base.cc:1116] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:52246
I20260806 09:21:43.566172 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 21822
I20260806 09:21:43.582665 22391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:43.567671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43774 (local address 127.16.86.129:42201)
0806 09:21:43.567926 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:43.567931 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:43.567975 (+ 44us) server_negotiation.cc:407] Connection header received
0806 09:21:43.568259 (+ 284us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:43.568267 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:43.568417 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:43.568537 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:43.570042 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.570799 (+ 757us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.572393 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.572975 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.573135 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:43.575887 (+ 2752us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:43.575894 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:43.575898 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:43.575949 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:43.579274 (+ 3325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:43.580213 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:43.580217 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:43.580220 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:43.580412 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:43.580927 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:43.580930 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:43.580933 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:43.581286 (+ 353us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:43.581514 (+ 228us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:43.581989 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 09:21:43.582285 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 09:21:43.583717 21994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:43.680801 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 21822
I20260806 09:21:43.711505 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 22008
Aug 06 09:21:43 dist-test-slave-h3ft krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:21:43.730991 22397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:43.712824 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.16.86.130:33635)
0806 09:21:43.713625 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:43.713634 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:43.713664 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:21:43.713734 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:43.713742 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:43.713981 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:43.714179 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:43.715691 (+ 1512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.716922 (+ 1231us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.718406 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.719300 (+ 894us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.719470 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:43.724661 (+ 5191us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:43.724675 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:43.724693 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:43.724747 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:43.727976 (+ 3229us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:43.728727 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:43.728736 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:43.728739 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:43.728864 (+ 125us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:43.729276 (+ 412us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:43.729281 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:43.729284 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:43.729659 (+ 375us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:43.730046 (+ 387us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:43.730326 (+ 280us) server_negotiation.cc:299] Negotiation successful
0806 09:21:43.730611 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":626,"thread_start_us":432,"threads_started":1}
I20260806 09:21:43.732764 22180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:43.828491 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 22008
I20260806 09:21:43.855540 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 22200
Aug 06 09:21:43 dist-test-slave-h3ft krb5kdc[21672](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:21:43.876184 22401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:43.856828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56428 (local address 127.16.86.131:37145)
0806 09:21:43.857552 (+ 724us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:43.857562 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:43.857602 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:21:43.857687 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:43.857695 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:43.857985 (+ 290us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:43.858151 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:43.859678 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.860781 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.862333 (+ 1552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:43.862955 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:43.863184 (+ 229us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:43.868695 (+ 5511us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:43.868714 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:43.868736 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:43.868790 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:43.872488 (+ 3698us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:43.873479 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:43.873488 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:43.873491 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:43.873651 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:43.874110 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:43.874113 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:43.874115 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:43.874573 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:43.874896 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:43.875392 (+ 496us) server_negotiation.cc:299] Negotiation successful
0806 09:21:43.875672 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":519,"thread_start_us":347,"threads_started":1}
I20260806 09:21:43.878212 22374 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:43.972380 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 22200
I20260806 09:21:44.000319 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 21693
I20260806 09:21:44.002213 21784 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:44.111227 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 21693
2026-08-06T09:21:44Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4827 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:21:44 dist-test-slave-h3ft krb5kdc[22409](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:21:44 dist-test-slave-h3ft krb5kdc[22409](info): set up 2 sockets
Aug 06 09:21:44 dist-test-slave-h3ft krb5kdc[22409](info): commencing operation
krb5kdc: starting...
W20260806 09:21:45.449040 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.000s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:21:45 dist-test-slave-h3ft krb5kdc[22409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:21:45Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:21:45.743848 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41219
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:41219
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:46.154093 22428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:46.154490 22428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:46.154596 22428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:46.162703 22428 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:46.162856 22428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:46.162909 22428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:46.162966 22428 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:46.163064 22428 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:21:46.174286 22428 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:41219
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41219
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:46.176007 22428 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:46.177598 22428 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:46.191781 22439 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:46.191893 22436 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:46.191893 22437 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:46.193462 22428 server_base.cc:1034] running on GCE node
I20260806 09:21:46.194483 22428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:46.196497 22428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:46.197757 22428 hybrid_clock.cc:648] HybridClock initialized: now 1786008106197703 us; error 48 us; skew 500 ppm
I20260806 09:21:46.198197 22428 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:46 dist-test-slave-h3ft krb5kdc[22409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008106, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:46.202284 22428 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:21:46.204038 22428 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:46.204197 22428 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:46.204623 22428 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:46.208411 22428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "f4cb499c810049259b88e9f673ff8105"
format_stamp: "Formatted at 2026-08-06 09:21:46 on dist-test-slave-h3ft"
server_key: "a823585189b12a68bbaca922c1500ae2"
server_key_iv: "f60b4e78151a5cdd366f906614f91aa3"
server_key_version: "encryptionkey@0"
I20260806 09:21:46.209352 22428 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "f4cb499c810049259b88e9f673ff8105"
format_stamp: "Formatted at 2026-08-06 09:21:46 on dist-test-slave-h3ft"
server_key: "a823585189b12a68bbaca922c1500ae2"
server_key_iv: "f60b4e78151a5cdd366f906614f91aa3"
server_key_version: "encryptionkey@0"
I20260806 09:21:46.216192 22428 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:21:46.220758 22449 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:46.222437 22428 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260806 09:21:46.222677 22428 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "f4cb499c810049259b88e9f673ff8105"
format_stamp: "Formatted at 2026-08-06 09:21:46 on dist-test-slave-h3ft"
server_key: "a823585189b12a68bbaca922c1500ae2"
server_key_iv: "f60b4e78151a5cdd366f906614f91aa3"
server_key_version: "encryptionkey@0"
I20260806 09:21:46.222980 22428 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:46.255453 22428 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:46.260715 22428 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:46.261058 22428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:46.284072 22428 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:41219
I20260806 09:21:46.284117 22530 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:41219 every 8 connection(s)
I20260806 09:21:46.285370 22428 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:46.289837 22531 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:46.295558 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 22428
I20260806 09:21:46.295887 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:46.296700 16730 external_mini_cluster.cc:1428] Setting key 8209727ba39b004291868308eb7a20c8
I20260806 09:21:46.301967 22531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105: Bootstrap starting.
I20260806 09:21:46.305537 22531 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:46.306619 22531 log.cc:824] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:46.310123 22531 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105: No bootstrap required, opened a new log
Aug 06 09:21:46 dist-test-slave-h3ft krb5kdc[22409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:46.319900 22531 raft_consensus.cc:348] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cb499c810049259b88e9f673ff8105" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41219 } }
I20260806 09:21:46.320423 22531 raft_consensus.cc:374] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:46.320546 22531 raft_consensus.cc:729] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f4cb499c810049259b88e9f673ff8105, State: Initialized, Role: FOLLOWER
I20260806 09:21:46.321447 22531 consensus_queue.cc:260] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cb499c810049259b88e9f673ff8105" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41219 } }
I20260806 09:21:46.321887 22531 raft_consensus.cc:388] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:46.322037 22531 raft_consensus.cc:482] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:46.322237 22531 raft_consensus.cc:3037] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:46.325649 22531 raft_consensus.cc:504] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cb499c810049259b88e9f673ff8105" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41219 } }
I20260806 09:21:46.326260 22531 leader_election.cc:302] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f4cb499c810049259b88e9f673ff8105; no voters:
I20260806 09:21:46.327169 22531 leader_election.cc:288] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:46.327178 22534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:46.299714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41528 (local address 127.16.86.190:41219)
0806 09:21:46.300842 (+ 1128us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:46.300857 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:46.300914 (+ 57us) server_negotiation.cc:407] Connection header received
0806 09:21:46.303452 (+ 2538us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:46.303491 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:46.304050 (+ 559us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:46.304594 (+ 544us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:46.306621 (+ 2027us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:46.308810 (+ 2189us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:46.310427 (+ 1617us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:46.311653 (+ 1226us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:46.311914 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:46.316900 (+ 4986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:46.316915 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:46.316943 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:46.316998 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:46.321634 (+ 4636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:46.322647 (+ 1013us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:46.322658 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:46.322670 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:46.322909 (+ 239us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:46.323370 (+ 461us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:46.323379 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:46.323383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:46.324179 (+ 796us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:46.324515 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:46.325017 (+ 502us) server_negotiation.cc:299] Negotiation successful
0806 09:21:46.325475 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":824,"thread_start_us":519,"threads_started":1}
I20260806 09:21:46.327525 22536 raft_consensus.cc:2781] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:46.328589 22536 raft_consensus.cc:686] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [term 1 LEADER]: Becoming Leader. State: Replica: f4cb499c810049259b88e9f673ff8105, State: Running, Role: LEADER
I20260806 09:21:46.329269 22536 consensus_queue.cc:237] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cb499c810049259b88e9f673ff8105" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41219 } }
I20260806 09:21:46.331431 22531 sys_catalog.cc:564] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:46.332769 22537 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f4cb499c810049259b88e9f673ff8105. Latest consensus state: current_term: 1 leader_uuid: "f4cb499c810049259b88e9f673ff8105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cb499c810049259b88e9f673ff8105" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41219 } } }
I20260806 09:21:46.333457 22537 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:46.333235 22539 sys_catalog.cc:455] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f4cb499c810049259b88e9f673ff8105" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f4cb499c810049259b88e9f673ff8105" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41219 } } }
I20260806 09:21:46.333870 22539 sys_catalog.cc:458] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:46.336444 22547 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:46.342500 22547 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:46.353960 22547 catalog_manager.cc:1347] Generated new cluster ID: 5b918d3248d44f1f8f4fc620d1bb822c
I20260806 09:21:46.354163 22547 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:46.384557 22547 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:46.386229 22547 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:46.393499 22547 catalog_manager.cc:5868] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105: Generated new TSK 0
I20260806 09:21:46.394593 22547 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:21:46.472425 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41219
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:46.898705 22570 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:46.899200 22570 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:46.899324 22570 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:46.907642 22570 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:46.907799 22570 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:46.907935 22570 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:21:46.919436 22570 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:46.921936 22570 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:46.924026 22570 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:46.941721 22580 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:46.941721 22584 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:46.942510 22588 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:46.943385 22570 server_base.cc:1034] running on GCE node
I20260806 09:21:46.944275 22570 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:46.945577 22570 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:46.946846 22570 hybrid_clock.cc:648] HybridClock initialized: now 1786008106946770 us; error 64 us; skew 500 ppm
I20260806 09:21:46.947304 22570 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:46 dist-test-slave-h3ft krb5kdc[22409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008106, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:46.951622 22570 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:21:46.953154 22570 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:46.953265 22570 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:46.953581 22570 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:46.957154 22570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9"
format_stamp: "Formatted at 2026-08-06 09:21:46 on dist-test-slave-h3ft"
server_key: "9e2d770bf8879c147ba2a486f80c1c19"
server_key_iv: "506bb5983bc02051e84292da36bdf026"
server_key_version: "encryptionkey@0"
I20260806 09:21:46.958056 22570 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9"
format_stamp: "Formatted at 2026-08-06 09:21:46 on dist-test-slave-h3ft"
server_key: "9e2d770bf8879c147ba2a486f80c1c19"
server_key_iv: "506bb5983bc02051e84292da36bdf026"
server_key_version: "encryptionkey@0"
I20260806 09:21:46.964444 22570 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 09:21:46.969012 22594 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:46.970947 22570 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:21:46.971208 22570 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9"
format_stamp: "Formatted at 2026-08-06 09:21:46 on dist-test-slave-h3ft"
server_key: "9e2d770bf8879c147ba2a486f80c1c19"
server_key_iv: "506bb5983bc02051e84292da36bdf026"
server_key_version: "encryptionkey@0"
I20260806 09:21:46.971475 22570 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:46.992261 22570 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:46.996903 22570 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:46.997239 22570 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:46.998430 22570 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:47.000025 22570 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:47.000161 22570 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:47.000315 22570 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:47.000416 22570 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:47.040817 22570 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:45613
I20260806 09:21:47.040936 22752 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:45613 every 8 connection(s)
I20260806 09:21:47.041929 22570 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:21:47.047070 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 22570
I20260806 09:21:47.047331 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:21:47.048264 16730 external_mini_cluster.cc:1428] Setting key b4075d21d2adb63e51888eacd2263633
Aug 06 09:21:47 dist-test-slave-h3ft krb5kdc[22409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008106, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:47.068753 22756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:47.045246 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:53799 (local address 127.16.86.190:41219)
0806 09:21:47.045816 (+ 570us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:47.045821 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:47.047154 (+ 1333us) server_negotiation.cc:407] Connection header received
0806 09:21:47.049706 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:47.049712 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:47.049817 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:47.049946 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:47.052098 (+ 2152us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:47.053138 (+ 1040us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:47.055939 (+ 2801us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:47.056401 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:47.056533 (+ 132us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:47.061628 (+ 5095us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:47.061634 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:47.061638 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:47.061678 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:47.064793 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:47.065837 (+ 1044us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:47.065842 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:47.065845 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:47.065988 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:47.066499 (+ 511us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:47.066503 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:47.066506 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:47.066862 (+ 356us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:47.067094 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:47.068028 (+ 934us) server_negotiation.cc:299] Negotiation successful
0806 09:21:47.068310 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":443,"thread_start_us":292,"threads_started":1}
I20260806 09:21:47.070066 22758 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:47.045889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41219 (local address 127.16.86.129:53799)
0806 09:21:47.046887 (+ 998us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:47.046953 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:47.049379 (+ 2426us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:47.050222 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:47.050231 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:47.050954 (+ 723us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:47.051861 (+ 907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:47.051890 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:47.053376 (+ 1486us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:47.053385 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:47.055748 (+ 2363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:47.055767 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:47.056697 (+ 930us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:47.056708 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:47.057052 (+ 344us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:47.057940 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:47.057977 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:47.061313 (+ 3336us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:47.065015 (+ 3702us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:47.065036 (+ 21us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:47.065057 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:47.065678 (+ 621us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:47.066120 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:47.066126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:47.066129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:47.066358 (+ 229us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:47.067048 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:47.067058 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:47.067547 (+ 489us) client_negotiation.cc:769] Sending connection context
0806 09:21:47.067882 (+ 335us) client_negotiation.cc:240] Negotiation successful
0806 09:21:47.068616 (+ 734us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":330,"threads_started":1}
I20260806 09:21:47.072355 22753 heartbeater.cc:347] Connected to a master server at 127.16.86.190:41219
I20260806 09:21:47.072793 22753 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:47.073686 22753 heartbeater.cc:511] Master 127.16.86.190:41219 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
I20260806 09:21:47.076692 22476 ts_manager.cc:194] Registered new tserver with Master: a1df2bcbeaac44f6b6d80f960b31f0f9 (127.16.86.129:45613)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
I20260806 09:21:47.078712 22476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:53799
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:21:47.111232 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41219
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:21:47.490346 22764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:47.490690 22764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:47.490797 22764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:47.498767 22764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:47.498921 22764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:47.499114 22764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:21:47.509477 22764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:47.510870 22764 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:47.512301 22764 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:47.526773 22771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:47.526856 22772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:47.526773 22774 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:47.528879 22764 server_base.cc:1034] running on GCE node
I20260806 09:21:47.529786 22764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:47.531384 22764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:47.532681 22764 hybrid_clock.cc:648] HybridClock initialized: now 1786008107532589 us; error 83 us; skew 500 ppm
I20260806 09:21:47.533217 22764 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:47 dist-test-slave-h3ft krb5kdc[22409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008107, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:47.542423 22764 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:21:47.544483 22764 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:47.544613 22764 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:47.545121 22764 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:47.549361 22764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "4456dad53fcc4fe886a60bdb1a7bcb96"
format_stamp: "Formatted at 2026-08-06 09:21:47 on dist-test-slave-h3ft"
server_key: "bc292f51f56305f9bb334387dd15577a"
server_key_iv: "1e97808c7f5a17aa45ac9ef93126c7f9"
server_key_version: "encryptionkey@0"
I20260806 09:21:47.550521 22764 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "4456dad53fcc4fe886a60bdb1a7bcb96"
format_stamp: "Formatted at 2026-08-06 09:21:47 on dist-test-slave-h3ft"
server_key: "bc292f51f56305f9bb334387dd15577a"
server_key_iv: "1e97808c7f5a17aa45ac9ef93126c7f9"
server_key_version: "encryptionkey@0"
I20260806 09:21:47.558065 22764 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260806 09:21:47.563381 22783 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:47.565572 22764 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260806 09:21:47.565822 22764 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "4456dad53fcc4fe886a60bdb1a7bcb96"
format_stamp: "Formatted at 2026-08-06 09:21:47 on dist-test-slave-h3ft"
server_key: "bc292f51f56305f9bb334387dd15577a"
server_key_iv: "1e97808c7f5a17aa45ac9ef93126c7f9"
server_key_version: "encryptionkey@0"
I20260806 09:21:47.566110 22764 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:47.598717 22764 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:47.605460 22764 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:47.605909 22764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:47.607450 22764 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:47.609408 22764 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:47.609551 22764 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:47.609714 22764 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:47.609814 22764 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:47.652802 22764 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:35185
I20260806 09:21:47.652930 22949 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:35185 every 8 connection(s)
I20260806 09:21:47.654045 22764 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:21:47.660822 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 22764
I20260806 09:21:47.661048 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:21:47.661545 16730 external_mini_cluster.cc:1428] Setting key 9603057bdf492fd3911969adf73f7d50
Aug 06 09:21:47 dist-test-slave-h3ft krb5kdc[22409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008107, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:47.679782 22952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:47.657439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:39721 (local address 127.16.86.190:41219)
0806 09:21:47.658050 (+ 611us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:47.658056 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:47.659259 (+ 1203us) server_negotiation.cc:407] Connection header received
0806 09:21:47.661669 (+ 2410us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:47.661676 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:47.661799 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:47.661977 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:47.664589 (+ 2612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:47.665412 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:47.667309 (+ 1897us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:47.667795 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:47.667963 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:47.672527 (+ 4564us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:47.672533 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:47.672536 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:47.672587 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:47.675763 (+ 3176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:47.676887 (+ 1124us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:47.676891 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:47.676894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:47.677036 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:47.677559 (+ 523us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:47.677564 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:47.677567 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:47.677913 (+ 346us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:47.678093 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:47.679054 (+ 961us) server_negotiation.cc:299] Negotiation successful
0806 09:21:47.679339 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":465,"thread_start_us":311,"threads_started":1}
I20260806 09:21:47.681084 22954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:47.658007 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41219 (local address 127.16.86.130:39721)
0806 09:21:47.658981 (+ 974us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:47.659103 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:47.661302 (+ 2199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:47.662322 (+ 1020us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:47.662337 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:47.663241 (+ 904us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:47.664317 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:47.664351 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:47.665573 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:47.665579 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:47.667142 (+ 1563us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:47.667157 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:47.668102 (+ 945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:47.668110 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:47.668399 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:47.669278 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:47.669312 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:47.672225 (+ 2913us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:47.675952 (+ 3727us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:47.675964 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:47.675990 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:47.676721 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:47.677163 (+ 442us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:47.677168 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:47.677172 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:47.677417 (+ 245us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:47.678054 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:47.678064 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:47.678557 (+ 493us) client_negotiation.cc:769] Sending connection context
0806 09:21:47.678900 (+ 343us) client_negotiation.cc:240] Negotiation successful
0806 09:21:47.679639 (+ 739us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":359,"threads_started":1}
I20260806 09:21:47.683128 22950 heartbeater.cc:347] Connected to a master server at 127.16.86.190:41219
I20260806 09:21:47.683670 22950 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:47.684778 22950 heartbeater.cc:511] Master 127.16.86.190:41219 requested a full tablet report, sending...
I20260806 09:21:47.687194 22476 ts_manager.cc:194] Registered new tserver with Master: 4456dad53fcc4fe886a60bdb1a7bcb96 (127.16.86.130:35185)
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
I20260806 09:21:47.688385 22476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:39721
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:21:47.727932 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41219
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:46337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:21:48.082763 22753 heartbeater.cc:503] Master 127.16.86.190:41219 was elected leader, sending a full tablet report...
W20260806 09:21:48.089417 22958 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:48.089740 22958 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:48.089843 22958 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:48.097218 22958 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:48.097334 22958 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:48.097499 22958 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:21:48.107955 22958 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41219
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:48.109395 22958 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:48.110781 22958 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:48.126852 22965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:48.126914 22964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:48.127314 22967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:48.128209 22958 server_base.cc:1034] running on GCE node
I20260806 09:21:48.129022 22958 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:48.130125 22958 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:48.131516 22958 hybrid_clock.cc:648] HybridClock initialized: now 1786008108131390 us; error 93 us; skew 500 ppm
I20260806 09:21:48.131948 22958 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:21:48 dist-test-slave-h3ft krb5kdc[22409](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008108, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:21:48.136164 22958 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:21:48.137574 22958 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:48.137718 22958 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:48.138069 22958 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:48.141266 22958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "821dd6af1eef4e7d81e7dca862a046aa"
format_stamp: "Formatted at 2026-08-06 09:21:48 on dist-test-slave-h3ft"
server_key: "ab1f028367f9fee761e080a3c4aee017"
server_key_iv: "192cfec3b0d665a7c20d95339925ec11"
server_key_version: "encryptionkey@0"
I20260806 09:21:48.142138 22958 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "821dd6af1eef4e7d81e7dca862a046aa"
format_stamp: "Formatted at 2026-08-06 09:21:48 on dist-test-slave-h3ft"
server_key: "ab1f028367f9fee761e080a3c4aee017"
server_key_iv: "192cfec3b0d665a7c20d95339925ec11"
server_key_version: "encryptionkey@0"
I20260806 09:21:48.148257 22958 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 09:21:48.152479 22975 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:48.154278 22958 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:21:48.154521 22958 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "821dd6af1eef4e7d81e7dca862a046aa"
format_stamp: "Formatted at 2026-08-06 09:21:48 on dist-test-slave-h3ft"
server_key: "ab1f028367f9fee761e080a3c4aee017"
server_key_iv: "192cfec3b0d665a7c20d95339925ec11"
server_key_version: "encryptionkey@0"
I20260806 09:21:48.154798 22958 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:48.183226 22958 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:48.187918 22958 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:48.188223 22958 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:48.189337 22958 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:48.191116 22958 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:48.191244 22958 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:48.191386 22958 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:48.191479 22958 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:48.232806 22958 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:34599
I20260806 09:21:48.232874 23144 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:34599 every 8 connection(s)
I20260806 09:21:48.233896 22958 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:21:48.241464 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 22958
I20260806 09:21:48.241731 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:21:48.242285 16730 external_mini_cluster.cc:1428] Setting key 813528a94dd3d4cd4bcaaa89ee84ca3d
Aug 06 09:21:48 dist-test-slave-h3ft krb5kdc[22409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008108, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:21:48.258471 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.237343 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:58207 (local address 127.16.86.190:41219)
0806 09:21:48.237976 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:48.237981 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:48.239262 (+ 1281us) server_negotiation.cc:407] Connection header received
0806 09:21:48.241304 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:48.241310 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:48.241433 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:48.241589 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:48.243701 (+ 2112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.244804 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.246631 (+ 1827us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.247159 (+ 528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.247304 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.251417 (+ 4113us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:48.251423 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:48.251426 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:48.251462 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:48.254617 (+ 3155us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:48.255706 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:48.255713 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:48.255722 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:48.255855 (+ 133us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:48.256337 (+ 482us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:48.256342 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:48.256344 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:48.256652 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:48.256834 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:48.257724 (+ 890us) server_negotiation.cc:299] Negotiation successful
0806 09:21:48.258011 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":498,"thread_start_us":336,"threads_started":1}
I20260806 09:21:48.259832 23150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.238006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41219 (local address 127.16.86.131:58207)
0806 09:21:48.238971 (+ 965us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:48.239093 (+ 122us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:48.240982 (+ 1889us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:48.241925 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:48.241938 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:48.242666 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:48.243488 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:48.243510 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.244988 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.244998 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:48.246461 (+ 1463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:48.246475 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.247426 (+ 951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.247434 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:48.247664 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.248474 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:21:48.248508 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:48.251163 (+ 2655us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:48.254792 (+ 3629us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:48.254804 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:48.254826 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:48.255560 (+ 734us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:48.255980 (+ 420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:21:48.255985 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:21:48.255989 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:21:48.256197 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:21:48.256798 (+ 601us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:48.256807 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:48.257261 (+ 454us) client_negotiation.cc:769] Sending connection context
0806 09:21:48.257626 (+ 365us) client_negotiation.cc:240] Negotiation successful
0806 09:21:48.258276 (+ 650us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":403,"threads_started":1}
I20260806 09:21:48.261845 23145 heartbeater.cc:347] Connected to a master server at 127.16.86.190:41219
I20260806 09:21:48.262229 23145 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:48.263162 23145 heartbeater.cc:511] Master 127.16.86.190:41219 requested a full tablet report, sending...
I20260806 09:21:48.265084 22476 ts_manager.cc:194] Registered new tserver with Master: 821dd6af1eef4e7d81e7dca862a046aa (127.16.86.131:34599)
I20260806 09:21:48.266018 22476 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:58207
I20260806 09:21:48.272833 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:21:48.294940 23147 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.278306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41530 (local address 127.16.86.190:41219)
0806 09:21:48.278613 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:48.278621 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:48.279277 (+ 656us) server_negotiation.cc:407] Connection header received
0806 09:21:48.279493 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:48.279498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:48.279587 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:48.279691 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:48.281238 (+ 1547us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.282091 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.283651 (+ 1560us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.284231 (+ 580us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.284366 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.287606 (+ 3240us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:48.287611 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:48.287615 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:48.287660 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:48.291361 (+ 3701us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:48.292294 (+ 933us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:48.292302 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:48.292305 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:48.292493 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:48.293019 (+ 526us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:48.293024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:48.293026 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:48.293476 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:48.293724 (+ 248us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:48.294200 (+ 476us) server_negotiation.cc:299] Negotiation successful
0806 09:21:48.294494 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 09:21:48.299824 22476 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41530:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:21:48.307201 22476 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:21:48.347680 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.323734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55258 (local address 127.16.86.131:34599)
0806 09:21:48.338731 (+ 14997us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:48.338745 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:48.338790 (+ 45us) server_negotiation.cc:407] Connection header received
0806 09:21:48.338941 (+ 151us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:48.338952 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:48.339470 (+ 518us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:48.340011 (+ 541us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:48.340731 (+ 720us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.341978 (+ 1247us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.344912 (+ 2934us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.345895 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.346054 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.346326 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:48.347111 (+ 785us) server_negotiation.cc:299] Negotiation successful
0806 09:21:48.347307 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14782,"thread_start_us":15541,"threads_started":1}
I20260806 09:21:48.347954 23167 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.323243 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:34599 (local address 127.0.0.1:55258)
0806 09:21:48.337187 (+ 13944us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:48.337208 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:48.337329 (+ 121us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:48.339766 (+ 2437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:48.339776 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:48.339824 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:48.340444 (+ 620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:48.340458 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.342160 (+ 1702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.342165 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:48.344742 (+ 2577us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:48.344762 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.346119 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.346125 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:48.346327 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.346392 (+ 65us) client_negotiation.cc:769] Sending connection context
0806 09:21:48.346561 (+ 169us) client_negotiation.cc:240] Negotiation successful
0806 09:21:48.346793 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13781,"mutex_wait_us":152,"thread_start_us":11774,"threads_started":1}
I20260806 09:21:48.349565 23176 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.324342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:35185 (local address 127.0.0.1:33900)
0806 09:21:48.338298 (+ 13956us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:48.338314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:48.338440 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:48.339516 (+ 1076us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:48.339524 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:48.339553 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:48.340290 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:48.340313 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.342615 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.342622 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:48.345890 (+ 3268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:48.345904 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.348586 (+ 2682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.348593 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:48.348805 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.348845 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 09:21:48.348941 (+ 96us) client_negotiation.cc:240] Negotiation successful
0806 09:21:48.349098 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13834,"mutex_wait_us":33,"thread_start_us":13679,"threads_started":1}
I20260806 09:21:48.350093 23168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.323172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:45613 (local address 127.0.0.1:48952)
0806 09:21:48.336176 (+ 13004us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:48.336238 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:48.336467 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:48.339111 (+ 2644us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:48.339122 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:48.339185 (+ 63us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:48.339847 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:48.339866 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.342409 (+ 2543us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.342423 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:48.345880 (+ 3457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:48.345904 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.348849 (+ 2945us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.348858 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:48.349106 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.349139 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 09:21:48.349274 (+ 135us) client_negotiation.cc:240] Negotiation successful
0806 09:21:48.349457 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12686,"mutex_wait_us":196,"thread_start_us":13647,"threads_started":1}
I20260806 09:21:48.350579 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.325040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33900 (local address 127.16.86.130:35185)
0806 09:21:48.337847 (+ 12807us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:48.337861 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:48.338563 (+ 702us) server_negotiation.cc:407] Connection header received
0806 09:21:48.338680 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:48.338688 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:48.338990 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:48.339235 (+ 245us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:48.340575 (+ 1340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.342378 (+ 1803us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.346170 (+ 3792us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.348092 (+ 1922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.348308 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.348579 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:48.349923 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0806 09:21:48.350177 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12637,"thread_start_us":12521,"threads_started":1}
I20260806 09:21:48.350585 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.323650 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48952 (local address 127.16.86.129:45613)
0806 09:21:48.337970 (+ 14320us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:48.337982 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:48.338032 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:21:48.338188 (+ 156us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:48.338204 (+ 16us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:48.338696 (+ 492us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:48.339626 (+ 930us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:48.340154 (+ 528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.342177 (+ 2023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.346161 (+ 3984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.348558 (+ 2397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.348739 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.349001 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:48.349359 (+ 358us) server_negotiation.cc:299] Negotiation successful
0806 09:21:48.349579 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14028,"thread_start_us":14137,"threads_started":1}
I20260806 09:21:48.352597 23050 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdc2de480d44364a5a78ed1a9ab676b (DEFAULT_TABLE table=test-table [id=3652df26ced14d43809cfd553d16710b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:48.354242 23050 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdc2de480d44364a5a78ed1a9ab676b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:48.354313 22854 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdc2de480d44364a5a78ed1a9ab676b (DEFAULT_TABLE table=test-table [id=3652df26ced14d43809cfd553d16710b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:48.355720 22854 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdc2de480d44364a5a78ed1a9ab676b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:48.355191 22658 tablet_service.cc:1467] Processing CreateTablet for tablet 9fdc2de480d44364a5a78ed1a9ab676b (DEFAULT_TABLE table=test-table [id=3652df26ced14d43809cfd553d16710b]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:48.356716 22658 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 9fdc2de480d44364a5a78ed1a9ab676b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:48.370407 23178 tablet_bootstrap.cc:492] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa: Bootstrap starting.
I20260806 09:21:48.372993 23179 tablet_bootstrap.cc:492] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96: Bootstrap starting.
I20260806 09:21:48.376890 23179 tablet_bootstrap.cc:654] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:48.377275 23178 tablet_bootstrap.cc:654] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:48.377421 23180 tablet_bootstrap.cc:492] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9: Bootstrap starting.
I20260806 09:21:48.378327 23179 log.cc:824] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:48.378641 23178 log.cc:824] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:48.381122 23180 tablet_bootstrap.cc:654] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:48.382136 23180 log.cc:824] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:48.382337 23178 tablet_bootstrap.cc:492] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa: No bootstrap required, opened a new log
I20260806 09:21:48.382341 23179 tablet_bootstrap.cc:492] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96: No bootstrap required, opened a new log
I20260806 09:21:48.382694 23178 ts_tablet_manager.cc:1397] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260806 09:21:48.382694 23179 ts_tablet_manager.cc:1397] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 09:21:48.385059 23180 tablet_bootstrap.cc:492] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9: No bootstrap required, opened a new log
I20260806 09:21:48.385361 23180 ts_tablet_manager.cc:1397] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.006s
I20260806 09:21:48.393193 23180 raft_consensus.cc:348] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45613 } } peers { permanent_uuid: "821dd6af1eef4e7d81e7dca862a046aa" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34599 } } peers { permanent_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35185 } }
I20260806 09:21:48.393599 23180 raft_consensus.cc:374] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:48.393692 23180 raft_consensus.cc:729] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a1df2bcbeaac44f6b6d80f960b31f0f9, State: Initialized, Role: FOLLOWER
I20260806 09:21:48.394361 23180 consensus_queue.cc:260] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45613 } } peers { permanent_uuid: "821dd6af1eef4e7d81e7dca862a046aa" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34599 } } peers { permanent_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35185 } }
I20260806 09:21:48.394793 23179 raft_consensus.cc:348] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45613 } } peers { permanent_uuid: "821dd6af1eef4e7d81e7dca862a046aa" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34599 } } peers { permanent_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35185 } }
I20260806 09:21:48.394791 23178 raft_consensus.cc:348] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45613 } } peers { permanent_uuid: "821dd6af1eef4e7d81e7dca862a046aa" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34599 } } peers { permanent_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35185 } }
I20260806 09:21:48.395403 23179 raft_consensus.cc:374] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:48.395424 23178 raft_consensus.cc:374] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:48.395529 23178 raft_consensus.cc:729] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 821dd6af1eef4e7d81e7dca862a046aa, State: Initialized, Role: FOLLOWER
I20260806 09:21:48.395529 23179 raft_consensus.cc:729] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4456dad53fcc4fe886a60bdb1a7bcb96, State: Initialized, Role: FOLLOWER
I20260806 09:21:48.396292 23180 ts_tablet_manager.cc:1428] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9: Time spent starting tablet: real 0.011s user 0.011s sys 0.000s
I20260806 09:21:48.396423 23178 consensus_queue.cc:260] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45613 } } peers { permanent_uuid: "821dd6af1eef4e7d81e7dca862a046aa" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34599 } } peers { permanent_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35185 } }
I20260806 09:21:48.396410 23179 consensus_queue.cc:260] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45613 } } peers { permanent_uuid: "821dd6af1eef4e7d81e7dca862a046aa" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34599 } } peers { permanent_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35185 } }
I20260806 09:21:48.398111 23145 heartbeater.cc:503] Master 127.16.86.190:41219 was elected leader, sending a full tablet report...
I20260806 09:21:48.399333 23178 ts_tablet_manager.cc:1428] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa: Time spent starting tablet: real 0.016s user 0.015s sys 0.004s
I20260806 09:21:48.402796 22950 heartbeater.cc:503] Master 127.16.86.190:41219 was elected leader, sending a full tablet report...
I20260806 09:21:48.403599 23179 ts_tablet_manager.cc:1428] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96: Time spent starting tablet: real 0.021s user 0.014s sys 0.009s
W20260806 09:21:48.406802 22951 tablet.cc:2369] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:21:48.462894 23185 raft_consensus.cc:482] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:21:48.463253 23185 raft_consensus.cc:504] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45613 } } peers { permanent_uuid: "821dd6af1eef4e7d81e7dca862a046aa" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34599 } } peers { permanent_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35185 } }
I20260806 09:21:48.465083 23185 leader_election.cc:288] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 821dd6af1eef4e7d81e7dca862a046aa (127.16.86.131:34599), 4456dad53fcc4fe886a60bdb1a7bcb96 (127.16.86.130:35185)
I20260806 09:21:48.473178 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.465629 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:49687 (local address 127.16.86.131:34599)
0806 09:21:48.466153 (+ 524us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:48.466160 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:48.466652 (+ 492us) server_negotiation.cc:407] Connection header received
0806 09:21:48.466912 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:48.466921 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:48.467061 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:48.467170 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:48.468008 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.468841 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.471096 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.472009 (+ 913us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.472147 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.472244 (+ 97us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:48.472758 (+ 514us) server_negotiation.cc:299] Negotiation successful
0806 09:21:48.472904 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336}
I20260806 09:21:48.473287 23194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.465286 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:34599 (local address 127.16.86.129:49687)
0806 09:21:48.466495 (+ 1209us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:48.466521 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:48.466744 (+ 223us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:48.467228 (+ 484us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:48.467234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:48.467265 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:48.467843 (+ 578us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:48.467855 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.468987 (+ 1132us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.468992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:48.470901 (+ 1909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:48.470914 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.472303 (+ 1389us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.472310 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:48.472515 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.472576 (+ 61us) client_negotiation.cc:769] Sending connection context
0806 09:21:48.472710 (+ 134us) client_negotiation.cc:240] Negotiation successful
0806 09:21:48.472894 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"thread_start_us":656,"threads_started":1}
I20260806 09:21:48.473764 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.465846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:57879 (local address 127.16.86.130:35185)
0806 09:21:48.466309 (+ 463us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:48.466317 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:48.467079 (+ 762us) server_negotiation.cc:407] Connection header received
0806 09:21:48.467316 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:48.467322 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:48.467430 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:48.467545 (+ 115us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:21:48.468283 (+ 738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.469194 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.471438 (+ 2244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.472635 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.472792 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.472922 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:48.473303 (+ 381us) server_negotiation.cc:299] Negotiation successful
0806 09:21:48.473473 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299}
I20260806 09:21:48.474220 23198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.465614 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:35185 (local address 127.16.86.129:57879)
0806 09:21:48.466904 (+ 1290us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:48.466926 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:48.467143 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:48.467578 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:48.467582 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:48.467603 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:21:48.468091 (+ 488us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:48.468106 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.469357 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.469361 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:48.471165 (+ 1804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:48.471180 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.472887 (+ 1707us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.472893 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:48.473082 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.473119 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 09:21:48.473242 (+ 123us) client_negotiation.cc:240] Negotiation successful
0806 09:21:48.473746 (+ 504us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1098,"mutex_wait_us":4,"thread_start_us":736,"threads_started":1}
I20260806 09:21:48.474807 23080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdc2de480d44364a5a78ed1a9ab676b" candidate_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "821dd6af1eef4e7d81e7dca862a046aa" is_pre_election: true
I20260806 09:21:48.475644 23080 raft_consensus.cc:2454] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1df2bcbeaac44f6b6d80f960b31f0f9 in term 0.
I20260806 09:21:48.475662 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdc2de480d44364a5a78ed1a9ab676b" candidate_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96" is_pre_election: true
I20260806 09:21:48.476305 22881 raft_consensus.cc:2454] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a1df2bcbeaac44f6b6d80f960b31f0f9 in term 0.
I20260806 09:21:48.476615 22595 leader_election.cc:302] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 821dd6af1eef4e7d81e7dca862a046aa, a1df2bcbeaac44f6b6d80f960b31f0f9; no voters:
I20260806 09:21:48.477074 23185 raft_consensus.cc:2781] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:21:48.477247 23185 raft_consensus.cc:482] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:21:48.477375 23185 raft_consensus.cc:3037] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:48.479825 23185 raft_consensus.cc:504] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45613 } } peers { permanent_uuid: "821dd6af1eef4e7d81e7dca862a046aa" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34599 } } peers { permanent_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35185 } }
I20260806 09:21:48.480487 23185 leader_election.cc:288] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [CANDIDATE]: Term 1 election: Requested vote from peers 821dd6af1eef4e7d81e7dca862a046aa (127.16.86.131:34599), 4456dad53fcc4fe886a60bdb1a7bcb96 (127.16.86.130:35185)
I20260806 09:21:48.481057 23080 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdc2de480d44364a5a78ed1a9ab676b" candidate_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "821dd6af1eef4e7d81e7dca862a046aa"
I20260806 09:21:48.481137 22881 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "9fdc2de480d44364a5a78ed1a9ab676b" candidate_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96"
I20260806 09:21:48.481359 23080 raft_consensus.cc:3037] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:48.481413 22881 raft_consensus.cc:3037] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:48.485144 23080 raft_consensus.cc:2454] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1df2bcbeaac44f6b6d80f960b31f0f9 in term 1.
I20260806 09:21:48.485141 22881 raft_consensus.cc:2454] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a1df2bcbeaac44f6b6d80f960b31f0f9 in term 1.
I20260806 09:21:48.485754 22598 leader_election.cc:302] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4456dad53fcc4fe886a60bdb1a7bcb96, a1df2bcbeaac44f6b6d80f960b31f0f9; no voters:
W20260806 09:21:48.485860 23146 tablet.cc:2369] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:21:48.486222 23185 raft_consensus.cc:2781] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:48.486804 23185 raft_consensus.cc:686] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 1 LEADER]: Becoming Leader. State: Replica: a1df2bcbeaac44f6b6d80f960b31f0f9, State: Running, Role: LEADER
I20260806 09:21:48.487427 23185 consensus_queue.cc:237] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45613 } } peers { permanent_uuid: "821dd6af1eef4e7d81e7dca862a046aa" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34599 } } peers { permanent_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35185 } }
I20260806 09:21:48.492184 22476 catalog_manager.cc:5495] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 reported cstate change: term changed from 0 to 1, leader changed from <none> to a1df2bcbeaac44f6b6d80f960b31f0f9 (127.16.86.129). New cstate: current_term: 1 leader_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "a1df2bcbeaac44f6b6d80f960b31f0f9" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45613 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "821dd6af1eef4e7d81e7dca862a046aa" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34599 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35185 } health_report { overall_health: UNKNOWN } } }
W20260806 09:21:48.545007 22755 tablet.cc:2369] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:21:48.546240 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.538934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36746 (local address 127.16.86.129:45613)
0806 09:21:48.539244 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:48.539254 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:48.539386 (+ 132us) server_negotiation.cc:407] Connection header received
0806 09:21:48.539609 (+ 223us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:48.539613 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:48.539726 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:48.539918 (+ 192us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:21:48.540354 (+ 436us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.541640 (+ 1286us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.543281 (+ 1641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.543810 (+ 529us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.543949 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.544595 (+ 646us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:21:48.545453 (+ 858us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:21:48.545645 (+ 192us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:48.545803 (+ 158us) server_negotiation.cc:299] Negotiation successful
0806 09:21:48.545917 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":110}
I20260806 09:21:48.556779 23080 raft_consensus.cc:1261] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa [term 1 FOLLOWER]: Refusing update from remote peer a1df2bcbeaac44f6b6d80f960b31f0f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:48.556779 22881 raft_consensus.cc:1261] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96 [term 1 FOLLOWER]: Refusing update from remote peer a1df2bcbeaac44f6b6d80f960b31f0f9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:48.558230 23199 consensus_queue.cc:1035] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4456dad53fcc4fe886a60bdb1a7bcb96" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35185 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:48.558764 23185 consensus_queue.cc:1035] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "821dd6af1eef4e7d81e7dca862a046aa" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34599 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:48.570858 23203 mvcc.cc:205] Tried to move back new op lower bound from 7315489212634013696 to 7315489212368556032. Current Snapshot: MvccSnapshot[applied={T|T < 7315489212634013696}]
I20260806 09:21:48.592371 22476 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:48.592761 22476 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:41530:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:48.597263 22476 catalog_manager.cc:5782] T 00000000000000000000000000000000 P f4cb499c810049259b88e9f673ff8105: Sending DeleteTablet for 3 replicas of tablet 9fdc2de480d44364a5a78ed1a9ab676b
I20260806 09:21:48.598740 22658 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdc2de480d44364a5a78ed1a9ab676b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:48 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:48952
I20260806 09:21:48.598732 23050 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdc2de480d44364a5a78ed1a9ab676b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:48 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:55258
I20260806 09:21:48.599071 22854 tablet_service.cc:1514] Processing DeleteTablet for tablet 9fdc2de480d44364a5a78ed1a9ab676b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:48 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:33900
I20260806 09:21:48.604167 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 22570
I20260806 09:21:48.608186 23217 tablet_replica.cc:330] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9: stopping tablet replica
I20260806 09:21:48.608870 23217 raft_consensus.cc:2229] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:21:48.609702 23217 raft_consensus.cc:2258] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:48.612347 23217 ts_tablet_manager.cc:1905] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:48.616546 23218 tablet_replica.cc:330] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96: stopping tablet replica
I20260806 09:21:48.616753 23215 tablet_replica.cc:330] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa: stopping tablet replica
I20260806 09:21:48.617259 23218 raft_consensus.cc:2229] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:48.617270 23215 raft_consensus.cc:2229] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:48.617853 23215 raft_consensus.cc:2258] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:48.617861 23218 raft_consensus.cc:2258] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:48.620522 23215 ts_tablet_manager.cc:1905] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:48.621367 23217 ts_tablet_manager.cc:1918] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:21:48.621596 23217 log.cc:1196] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/9fdc2de480d44364a5a78ed1a9ab676b
I20260806 09:21:48.622267 23217 ts_tablet_manager.cc:1939] T 9fdc2de480d44364a5a78ed1a9ab676b P a1df2bcbeaac44f6b6d80f960b31f0f9: Deleting consensus metadata
I20260806 09:21:48.623450 23218 ts_tablet_manager.cc:1905] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:48.625047 22454 catalog_manager.cc:4841] TS a1df2bcbeaac44f6b6d80f960b31f0f9 (127.16.86.129:45613): tablet 9fdc2de480d44364a5a78ed1a9ab676b (table test-table [id=3652df26ced14d43809cfd553d16710b]) successfully deleted
I20260806 09:21:48.627774 23215 ts_tablet_manager.cc:1918] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:21:48.627954 23215 log.cc:1196] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/9fdc2de480d44364a5a78ed1a9ab676b
I20260806 09:21:48.628571 23215 ts_tablet_manager.cc:1939] T 9fdc2de480d44364a5a78ed1a9ab676b P 821dd6af1eef4e7d81e7dca862a046aa: Deleting consensus metadata
I20260806 09:21:48.630515 22451 catalog_manager.cc:4841] TS 821dd6af1eef4e7d81e7dca862a046aa (127.16.86.131:34599): tablet 9fdc2de480d44364a5a78ed1a9ab676b (table test-table [id=3652df26ced14d43809cfd553d16710b]) successfully deleted
Aug 06 09:21:48 dist-test-slave-h3ft krb5kdc[22409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:21:48.633276 23218 ts_tablet_manager.cc:1918] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:21:48.633461 23218 log.cc:1196] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/9fdc2de480d44364a5a78ed1a9ab676b
I20260806 09:21:48.633972 23218 ts_tablet_manager.cc:1939] T 9fdc2de480d44364a5a78ed1a9ab676b P 4456dad53fcc4fe886a60bdb1a7bcb96: Deleting consensus metadata
I20260806 09:21:48.635589 22455 catalog_manager.cc:4841] TS 4456dad53fcc4fe886a60bdb1a7bcb96 (127.16.86.130:35185): tablet 9fdc2de480d44364a5a78ed1a9ab676b (table test-table [id=3652df26ced14d43809cfd553d16710b]) successfully deleted
I20260806 09:21:48.641742 23172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.607849 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36748 (local address 127.16.86.129:45613)
0806 09:21:48.608152 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:48.608173 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:48.609966 (+ 1793us) server_negotiation.cc:407] Connection header received
0806 09:21:48.611700 (+ 1734us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:48.611710 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:48.611874 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:48.611981 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:48.618369 (+ 6388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.619785 (+ 1416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.623772 (+ 3987us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.624638 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.624753 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.632510 (+ 7757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:48.632533 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:48.632574 (+ 41us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:48.632661 (+ 87us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:48.638145 (+ 5484us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:48.639093 (+ 948us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:48.639105 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:48.639109 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:48.639336 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:48.639753 (+ 417us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:48.639759 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:48.639762 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:48.640373 (+ 611us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:48.640734 (+ 361us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:48.640948 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 09:21:48.641268 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":158}
I20260806 09:21:48.642627 22745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:48.759658 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 22570
I20260806 09:21:48.791709 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 22764
Aug 06 09:21:48 dist-test-slave-h3ft krb5kdc[22409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:21:48.810315 23177 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.792760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39178 (local address 127.16.86.130:35185)
0806 09:21:48.793021 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:48.793030 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:48.793054 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:21:48.793283 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:48.793290 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:48.793382 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:48.793510 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:48.794901 (+ 1391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.795976 (+ 1075us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.797341 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.797788 (+ 447us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.797944 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.802863 (+ 4919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:48.802886 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:48.802915 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:48.802962 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:48.806658 (+ 3696us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:48.807740 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:48.807749 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:48.807752 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:48.807921 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:48.808400 (+ 479us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:48.808406 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:48.808409 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:48.808886 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:48.809271 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:48.809612 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 09:21:48.809893 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 09:21:48.811452 22943 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:48.921810 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 22764
I20260806 09:21:48.947927 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 22958
Aug 06 09:21:48 dist-test-slave-h3ft krb5kdc[22409](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008105, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:21:48.967151 23171 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:48.948969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41360 (local address 127.16.86.131:34599)
0806 09:21:48.949174 (+ 205us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:48.949179 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:48.949269 (+ 90us) server_negotiation.cc:407] Connection header received
0806 09:21:48.949440 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:48.949444 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:48.949541 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:48.949652 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:48.951520 (+ 1868us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.952260 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.953871 (+ 1611us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:48.954279 (+ 408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:48.954384 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:48.960447 (+ 6063us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:48.960467 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:48.960485 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:21:48.960529 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:48.963677 (+ 3148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:48.964753 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:48.964760 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:48.964763 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:48.964921 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:21:48.965395 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:21:48.965399 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:21:48.965401 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:21:48.965871 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:48.966153 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:48.966513 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 09:21:48.966767 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 09:21:48.967935 23136 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:49.070026 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 22958
I20260806 09:21:49.095546 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 22428
I20260806 09:21:49.097637 22525 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:49.212728 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 22428
2026-08-06T09:21:49Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5099 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2026-08-06T09:21:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:49Z Disabled control of system clock
I20260806 09:21:49.312918 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:43015
--webserver_interface=127.16.86.190
--webserver_port=0
--builtin_ntp_servers=127.16.86.148:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:43015
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 09:21:49.709834 23232 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:49.710158 23232 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:49.710285 23232 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:49.718000 23232 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:49.718124 23232 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:49.718164 23232 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:49.718282 23232 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:49.718355 23232 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:21:49.727988 23232 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:43015
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:43015
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:49.729455 23232 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:49.731524 23232 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:49.745265 23247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:49.745319 23241 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:49.745411 23239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:49.747107 23232 server_base.cc:1034] running on GCE node
I20260806 09:21:49.748267 23232 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:49.750155 23232 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:49.751418 23232 hybrid_clock.cc:648] HybridClock initialized: now 1786008109751367 us; error 41 us; skew 500 ppm
I20260806 09:21:49.751859 23232 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:21:49.754798 23232 webserver.cc:460] Webserver started at http://127.16.86.190:43447/ using document root <none> and password file <none>
I20260806 09:21:49.755764 23232 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:49.755913 23232 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:49.756326 23232 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:49.760013 23232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "be0317b2566646ce8bc0956993ccc465"
format_stamp: "Formatted at 2026-08-06 09:21:49 on dist-test-slave-h3ft"
server_key: "b812071856099d27e3c1ffa96c7a58a3"
server_key_iv: "c23154b88bd06e6e666d25ab7287c3f3"
server_key_version: "encryptionkey@0"
I20260806 09:21:49.761003 23232 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "be0317b2566646ce8bc0956993ccc465"
format_stamp: "Formatted at 2026-08-06 09:21:49 on dist-test-slave-h3ft"
server_key: "b812071856099d27e3c1ffa96c7a58a3"
server_key_iv: "c23154b88bd06e6e666d25ab7287c3f3"
server_key_version: "encryptionkey@0"
I20260806 09:21:49.767819 23232 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:21:49.772156 23256 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:49.773886 23232 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:21:49.774158 23232 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "be0317b2566646ce8bc0956993ccc465"
format_stamp: "Formatted at 2026-08-06 09:21:49 on dist-test-slave-h3ft"
server_key: "b812071856099d27e3c1ffa96c7a58a3"
server_key_iv: "c23154b88bd06e6e666d25ab7287c3f3"
server_key_version: "encryptionkey@0"
I20260806 09:21:49.774433 23232 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:49.781698 23232 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:49.782804 23232 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:49.783264 23232 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:49.806493 23232 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:43015
I20260806 09:21:49.806528 23330 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:43015 every 8 connection(s)
I20260806 09:21:49.808339 23232 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:49.813046 23331 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:49.815503 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 23232
I20260806 09:21:49.815740 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:49.816627 16730 external_mini_cluster.cc:1428] Setting key 92382d327c23b70dc9ebd58346507289
I20260806 09:21:49.827498 23335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:49.819580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35306 (local address 127.16.86.190:43015)
0806 09:21:49.820933 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:49.820947 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:49.821003 (+ 56us) server_negotiation.cc:407] Connection header received
0806 09:21:49.822983 (+ 1980us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:49.823087 (+ 104us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:49.823663 (+ 576us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:49.824116 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:49.825420 (+ 1304us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:49.825428 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:49.825442 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:49.825458 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:49.825545 (+ 87us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:49.825570 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:49.825791 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:49.825899 (+ 108us) server_negotiation.cc:299] Negotiation successful
0806 09:21:49.826256 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":497,"threads_started":1}
I20260806 09:21:49.827625 23331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465: Bootstrap starting.
I20260806 09:21:49.831115 23331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:49.832180 23331 log.cc:824] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:49.835451 23331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465: No bootstrap required, opened a new log
I20260806 09:21:49.843365 23331 raft_consensus.cc:348] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0317b2566646ce8bc0956993ccc465" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43015 } }
I20260806 09:21:49.843796 23331 raft_consensus.cc:374] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:49.843905 23331 raft_consensus.cc:729] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: be0317b2566646ce8bc0956993ccc465, State: Initialized, Role: FOLLOWER
I20260806 09:21:49.844558 23331 consensus_queue.cc:260] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0317b2566646ce8bc0956993ccc465" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43015 } }
I20260806 09:21:49.844826 23331 raft_consensus.cc:388] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:49.844973 23331 raft_consensus.cc:482] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:49.845160 23331 raft_consensus.cc:3037] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:49.847478 23331 raft_consensus.cc:504] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0317b2566646ce8bc0956993ccc465" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43015 } }
I20260806 09:21:49.848115 23331 leader_election.cc:302] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: be0317b2566646ce8bc0956993ccc465; no voters:
I20260806 09:21:49.848989 23331 leader_election.cc:288] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:49.849280 23338 raft_consensus.cc:2781] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:49.850319 23338 raft_consensus.cc:686] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [term 1 LEADER]: Becoming Leader. State: Replica: be0317b2566646ce8bc0956993ccc465, State: Running, Role: LEADER
I20260806 09:21:49.851171 23338 consensus_queue.cc:237] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0317b2566646ce8bc0956993ccc465" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43015 } }
I20260806 09:21:49.851450 23331 sys_catalog.cc:564] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:49.855154 23339 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "be0317b2566646ce8bc0956993ccc465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0317b2566646ce8bc0956993ccc465" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43015 } } }
I20260806 09:21:49.855463 23340 sys_catalog.cc:455] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [sys.catalog]: SysCatalogTable state changed. Reason: New leader be0317b2566646ce8bc0956993ccc465. Latest consensus state: current_term: 1 leader_uuid: "be0317b2566646ce8bc0956993ccc465" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "be0317b2566646ce8bc0956993ccc465" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43015 } } }
I20260806 09:21:49.855846 23339 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:49.855890 23340 sys_catalog.cc:458] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:49.859336 23350 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:49.865630 23350 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:49.885968 23350 catalog_manager.cc:1347] Generated new cluster ID: 9e40de227ff7452381ad3738f1cd02c0
I20260806 09:21:49.886260 23350 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:49.899845 23350 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:49.900359 23350 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:49.908514 23350 catalog_manager.cc:5868] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465: Generated new TSK 0
I20260806 09:21:49.909493 23350 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:21:49.924263 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43015
--builtin_ntp_servers=127.16.86.148:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20260806 09:21:50.299690 23371 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:50.299988 23371 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:50.300132 23371 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:50.307680 23371 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:50.307792 23371 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:50.307965 23371 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:21:50.317582 23371 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:50.318919 23371 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:50.320744 23371 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:50.335512 23381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:50.335592 23384 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:50.335744 23386 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:50.336930 23371 server_base.cc:1034] running on GCE node
I20260806 09:21:50.337563 23371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:50.338586 23371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:50.339871 23371 hybrid_clock.cc:648] HybridClock initialized: now 1786008110339808 us; error 53 us; skew 500 ppm
I20260806 09:21:50.340266 23371 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:21:50.342752 23371 webserver.cc:460] Webserver started at http://127.16.86.129:32915/ using document root <none> and password file <none>
I20260806 09:21:50.343556 23371 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:50.343655 23371 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:50.343955 23371 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:50.346868 23371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "254fbe73f1f24ba196bb46becf8ef6b6"
format_stamp: "Formatted at 2026-08-06 09:21:50 on dist-test-slave-h3ft"
server_key: "f03165a69b59936d92896bb7faf6cf7d"
server_key_iv: "3f39004eea7822c961dd44e56a9c9542"
server_key_version: "encryptionkey@0"
I20260806 09:21:50.347749 23371 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "254fbe73f1f24ba196bb46becf8ef6b6"
format_stamp: "Formatted at 2026-08-06 09:21:50 on dist-test-slave-h3ft"
server_key: "f03165a69b59936d92896bb7faf6cf7d"
server_key_iv: "3f39004eea7822c961dd44e56a9c9542"
server_key_version: "encryptionkey@0"
I20260806 09:21:50.353677 23371 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:21:50.357797 23393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:50.359623 23371 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260806 09:21:50.359794 23371 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "254fbe73f1f24ba196bb46becf8ef6b6"
format_stamp: "Formatted at 2026-08-06 09:21:50 on dist-test-slave-h3ft"
server_key: "f03165a69b59936d92896bb7faf6cf7d"
server_key_iv: "3f39004eea7822c961dd44e56a9c9542"
server_key_version: "encryptionkey@0"
I20260806 09:21:50.360000 23371 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:50.366302 23371 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:50.367249 23371 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:50.367542 23371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:50.368610 23371 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:50.370136 23371 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:50.370229 23371 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:50.370393 23371 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:50.370497 23371 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:50.410291 23371 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:46383
I20260806 09:21:50.410401 23552 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:46383 every 8 connection(s)
I20260806 09:21:50.411861 23371 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:21:50.415138 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 23371
I20260806 09:21:50.415395 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:21:50.416317 16730 external_mini_cluster.cc:1428] Setting key da1b4f8cb173b947b8a3419dd0dce557
I20260806 09:21:50.419353 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43015
--builtin_ntp_servers=127.16.86.148:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 09:21:50.422745 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:50.415406 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:58879 (local address 127.16.86.190:43015)
0806 09:21:50.416089 (+ 683us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:50.416098 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:50.417861 (+ 1763us) server_negotiation.cc:407] Connection header received
0806 09:21:50.419992 (+ 2131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:50.419998 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:50.420094 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:50.420232 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:50.421840 (+ 1608us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:50.421846 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:50.421852 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:50.421872 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:50.421929 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:50.421942 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:50.422098 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:50.422300 (+ 202us) server_negotiation.cc:299] Negotiation successful
0806 09:21:50.422383 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":337,"threads_started":1}
I20260806 09:21:50.423859 23561 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:50.416240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43015 (local address 127.16.86.129:58879)
0806 09:21:50.417633 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:50.417699 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:50.419670 (+ 1971us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:50.420515 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:50.420531 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:50.420638 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:50.421499 (+ 861us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:21:50.421502 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:50.421554 (+ 52us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:21:50.421655 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:50.422053 (+ 398us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:50.422065 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:50.422075 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:21:50.422399 (+ 324us) client_negotiation.cc:240] Negotiation successful
0806 09:21:50.422709 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":464,"threads_started":1}
I20260806 09:21:50.425755 23553 heartbeater.cc:347] Connected to a master server at 127.16.86.190:43015
I20260806 09:21:50.426164 23553 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:50.426892 23553 heartbeater.cc:511] Master 127.16.86.190:43015 requested a full tablet report, sending...
I20260806 09:21:50.429466 23282 ts_manager.cc:194] Registered new tserver with Master: 254fbe73f1f24ba196bb46becf8ef6b6 (127.16.86.129:46383)
W20260806 09:21:50.775717 23562 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:50.776083 23562 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:50.776279 23562 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:50.784789 23562 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:50.784929 23562 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:50.785107 23562 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:21:50.794732 23562 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:50.796146 23562 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:50.798046 23562 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:50.814446 23569 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:50.814636 23568 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:50.815526 23571 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:50.816327 23562 server_base.cc:1034] running on GCE node
I20260806 09:21:50.817273 23562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:50.818753 23562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:50.820099 23562 hybrid_clock.cc:648] HybridClock initialized: now 1786008110820020 us; error 61 us; skew 500 ppm
I20260806 09:21:50.820567 23562 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:21:50.823817 23562 webserver.cc:460] Webserver started at http://127.16.86.130:45869/ using document root <none> and password file <none>
I20260806 09:21:50.824793 23562 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:50.824960 23562 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:50.825385 23562 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:50.829115 23562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "5ca349add8c241d3880f827976ddb553"
format_stamp: "Formatted at 2026-08-06 09:21:50 on dist-test-slave-h3ft"
server_key: "71417e8d2aec99f40e0aab431eb9535b"
server_key_iv: "7158fc4d0c6583af357161aadd7f458d"
server_key_version: "encryptionkey@0"
I20260806 09:21:50.830240 23562 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "5ca349add8c241d3880f827976ddb553"
format_stamp: "Formatted at 2026-08-06 09:21:50 on dist-test-slave-h3ft"
server_key: "71417e8d2aec99f40e0aab431eb9535b"
server_key_iv: "7158fc4d0c6583af357161aadd7f458d"
server_key_version: "encryptionkey@0"
I20260806 09:21:50.837379 23562 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.003s
I20260806 09:21:50.842619 23580 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:50.845347 23562 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260806 09:21:50.845692 23562 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "5ca349add8c241d3880f827976ddb553"
format_stamp: "Formatted at 2026-08-06 09:21:50 on dist-test-slave-h3ft"
server_key: "71417e8d2aec99f40e0aab431eb9535b"
server_key_iv: "7158fc4d0c6583af357161aadd7f458d"
server_key_version: "encryptionkey@0"
I20260806 09:21:50.846077 23562 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:50.855839 23562 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:50.857149 23562 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:50.857507 23562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:50.858847 23562 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:50.860854 23562 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:50.860970 23562 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:50.861159 23562 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:50.861251 23562 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:50.905467 23562 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:35687
I20260806 09:21:50.905594 23746 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:35687 every 8 connection(s)
I20260806 09:21:50.907402 23562 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:21:50.917868 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 23562
I20260806 09:21:50.918087 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:21:50.918807 16730 external_mini_cluster.cc:1428] Setting key 5b6b54a700c6b3de2420816934937971
I20260806 09:21:50.918946 23560 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:50.911137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:32979 (local address 127.16.86.190:43015)
0806 09:21:50.911404 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:50.911412 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:50.913229 (+ 1817us) server_negotiation.cc:407] Connection header received
0806 09:21:50.915735 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:50.915743 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:50.915862 (+ 119us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:50.916012 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:50.917830 (+ 1818us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:50.917837 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:50.917841 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:50.917861 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:50.917909 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:50.917924 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:50.918115 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:50.918548 (+ 433us) server_negotiation.cc:299] Negotiation successful
0806 09:21:50.918624 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 09:21:50.920486 23754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:50.911689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43015 (local address 127.16.86.130:32979)
0806 09:21:50.912984 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:50.913046 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:50.915361 (+ 2315us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:50.916327 (+ 966us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:50.916337 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:50.916481 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:50.917428 (+ 947us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:21:50.917437 (+ 9us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:50.917487 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:21:50.917608 (+ 121us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:50.918156 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:50.918177 (+ 21us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:50.918191 (+ 14us) client_negotiation.cc:769] Sending connection context
0806 09:21:50.918678 (+ 487us) client_negotiation.cc:240] Negotiation successful
0806 09:21:50.919070 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":624,"thread_start_us":373,"threads_started":1}
I20260806 09:21:50.922247 23748 heartbeater.cc:347] Connected to a master server at 127.16.86.190:43015
I20260806 09:21:50.922621 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43015
--builtin_ntp_servers=127.16.86.148:42503
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20260806 09:21:50.922708 23748 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:50.923590 23748 heartbeater.cc:511] Master 127.16.86.190:43015 requested a full tablet report, sending...
I20260806 09:21:50.925612 23282 ts_manager.cc:194] Registered new tserver with Master: 5ca349add8c241d3880f827976ddb553 (127.16.86.130:35687)
W20260806 09:21:51.346566 23755 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:51.346987 23755 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:51.347215 23755 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:51.355783 23755 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:51.355942 23755 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:51.356151 23755 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:21:51.367218 23755 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42503
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43015
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:51.369086 23755 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:51.371560 23755 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:51.387713 23762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:51.388015 23770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:51.388584 23765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:51.389453 23755 server_base.cc:1034] running on GCE node
I20260806 09:21:51.390291 23755 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:51.391691 23755 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:51.392966 23755 hybrid_clock.cc:648] HybridClock initialized: now 1786008111392898 us; error 48 us; skew 500 ppm
I20260806 09:21:51.393447 23755 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:21:51.396771 23755 webserver.cc:460] Webserver started at http://127.16.86.131:35359/ using document root <none> and password file <none>
I20260806 09:21:51.397800 23755 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:51.397951 23755 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:51.398419 23755 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:51.402400 23755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1"
format_stamp: "Formatted at 2026-08-06 09:21:51 on dist-test-slave-h3ft"
server_key: "24d29559ee3b0c346c7b3075e82bc471"
server_key_iv: "6f2a3815d62eef5442bebea3293ebbfa"
server_key_version: "encryptionkey@0"
I20260806 09:21:51.403537 23755 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1"
format_stamp: "Formatted at 2026-08-06 09:21:51 on dist-test-slave-h3ft"
server_key: "24d29559ee3b0c346c7b3075e82bc471"
server_key_iv: "6f2a3815d62eef5442bebea3293ebbfa"
server_key_version: "encryptionkey@0"
I20260806 09:21:51.411528 23755 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260806 09:21:51.417891 23779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:51.420148 23755 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 09:21:51.420431 23755 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1"
format_stamp: "Formatted at 2026-08-06 09:21:51 on dist-test-slave-h3ft"
server_key: "24d29559ee3b0c346c7b3075e82bc471"
server_key_iv: "6f2a3815d62eef5442bebea3293ebbfa"
server_key_version: "encryptionkey@0"
I20260806 09:21:51.420714 23755 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:51.428658 23755 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:51.429783 23755 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:51.430173 23755 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:51.431529 23755 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:21:51.433183 23553 heartbeater.cc:503] Master 127.16.86.190:43015 was elected leader, sending a full tablet report...
I20260806 09:21:51.433619 23755 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:21:51.433736 23755 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:51.433933 23755 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:21:51.434056 23755 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:51.479501 23755 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:34781
I20260806 09:21:51.479529 23953 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:34781 every 8 connection(s)
I20260806 09:21:51.481348 23755 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:21:51.486672 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 23755
I20260806 09:21:51.486876 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:21:51.487499 16730 external_mini_cluster.cc:1428] Setting key 0ef8bf73c411261e46511a5fc201ee5b
I20260806 09:21:51.492560 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.484863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:46989 (local address 127.16.86.190:43015)
0806 09:21:51.485484 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:51.485491 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:51.487472 (+ 1981us) server_negotiation.cc:407] Connection header received
0806 09:21:51.489874 (+ 2402us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:51.489880 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:51.489958 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:51.490080 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:51.491772 (+ 1692us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:51.491777 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:51.491781 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:51.491799 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:51.491849 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:51.491863 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:51.491976 (+ 113us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:51.492189 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 09:21:51.492251 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":448,"thread_start_us":288,"threads_started":1}
I20260806 09:21:51.493594 23958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.485756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43015 (local address 127.16.86.131:46989)
0806 09:21:51.487235 (+ 1479us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:51.487299 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:51.489509 (+ 2210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:51.490361 (+ 852us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:51.490369 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:51.490476 (+ 107us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:51.491384 (+ 908us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:21:51.491387 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:51.491437 (+ 50us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:21:51.491575 (+ 138us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:51.491969 (+ 394us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:51.491978 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:51.491983 (+ 5us) client_negotiation.cc:769] Sending connection context
0806 09:21:51.492241 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 09:21:51.492466 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":683,"thread_start_us":397,"threads_started":1}
I20260806 09:21:51.495500 23954 heartbeater.cc:347] Connected to a master server at 127.16.86.190:43015
I20260806 09:21:51.495858 23954 heartbeater.cc:464] Registering TS with master...
I20260806 09:21:51.496528 23954 heartbeater.cc:511] Master 127.16.86.190:43015 requested a full tablet report, sending...
I20260806 09:21:51.498190 23282 ts_manager.cc:194] Registered new tserver with Master: 52b7a22fad9b49f6b3729a5fe2d45fc1 (127.16.86.131:34781)
I20260806 09:21:51.504678 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:21:51.513223 23957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.510120 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.16.86.190:43015)
0806 09:21:51.510428 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:51.510434 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:51.510897 (+ 463us) server_negotiation.cc:407] Connection header received
0806 09:21:51.511122 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:51.511127 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:51.511199 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:51.511317 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:51.512497 (+ 1180us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:51.512502 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:51.512506 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:51.512524 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:51.512576 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:51.512591 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:51.512721 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:51.512873 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 09:21:51.512940 (+ 67us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106}
I20260806 09:21:51.518239 23282 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35322:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:21:51.525393 23282 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:21:51.548934 23971 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.540894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:46383 (local address 127.0.0.1:51492)
0806 09:21:51.542202 (+ 1308us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:51.542262 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:51.542423 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:51.545006 (+ 2583us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:51.545019 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:51.545170 (+ 151us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:51.546238 (+ 1068us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:21:51.546242 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:51.546280 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:21:51.546389 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:51.547970 (+ 1581us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:51.547985 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:51.547995 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:21:51.548188 (+ 193us) client_negotiation.cc:240] Negotiation successful
0806 09:21:51.548310 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1007,"thread_start_us":830,"threads_started":1}
I20260806 09:21:51.549712 23977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.542727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:35687 (local address 127.0.0.1:33632)
0806 09:21:51.545157 (+ 2430us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:51.545192 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:51.545370 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:51.547081 (+ 1711us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:51.547089 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:51.547119 (+ 30us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:51.548110 (+ 991us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:21:51.548114 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:51.548133 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:21:51.548186 (+ 53us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:51.549020 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:51.549035 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:51.549041 (+ 6us) client_negotiation.cc:769] Sending connection context
0806 09:21:51.549174 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 09:21:51.549234 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2222,"thread_start_us":2045,"threads_started":1}
I20260806 09:21:51.550063 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.541268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51492 (local address 127.16.86.129:46383)
0806 09:21:51.544098 (+ 2830us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:51.544115 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:51.544162 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:21:51.544267 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:51.544281 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:51.544676 (+ 395us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:51.545882 (+ 1206us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:51.547659 (+ 1777us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:51.547671 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:51.547693 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:51.547717 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:51.547793 (+ 76us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:51.547820 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:51.549334 (+ 1514us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:51.549420 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 09:21:51.549573 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1955,"thread_start_us":1738,"threads_started":1}
I20260806 09:21:51.550678 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.543399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33632 (local address 127.16.86.130:35687)
0806 09:21:51.546403 (+ 3004us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:51.546411 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:51.546441 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:21:51.546525 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:51.546531 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:51.546802 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:51.546981 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:51.548721 (+ 1740us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:51.548735 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:51.548753 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:51.548776 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:51.548851 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:51.548874 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:51.550102 (+ 1228us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:51.550188 (+ 86us) server_negotiation.cc:299] Negotiation successful
0806 09:21:51.550303 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2792,"thread_start_us":2631,"threads_started":1}
I20260806 09:21:51.550746 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.542245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50630 (local address 127.16.86.131:34781)
0806 09:21:51.546389 (+ 4144us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:51.546400 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:51.546441 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:21:51.546522 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:51.546531 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:51.546802 (+ 271us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:51.546984 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:51.549482 (+ 2498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:51.549489 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:51.549501 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:51.549517 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:51.549567 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:51.549583 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:51.549797 (+ 214us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:51.550192 (+ 395us) server_negotiation.cc:299] Negotiation successful
0806 09:21:51.550321 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3990,"thread_start_us":3431,"threads_started":1}
I20260806 09:21:51.551098 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.541378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:34781 (local address 127.0.0.1:50630)
0806 09:21:51.543616 (+ 2238us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:51.543649 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:51.543798 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:51.547558 (+ 3760us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:51.547565 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:51.547589 (+ 24us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:51.548441 (+ 852us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:21:51.548443 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:51.548458 (+ 15us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:21:51.548496 (+ 38us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:51.549728 (+ 1232us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:51.549734 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:51.549742 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 09:21:51.550303 (+ 561us) client_negotiation.cc:240] Negotiation successful
0806 09:21:51.550376 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2056,"thread_start_us":1346,"threads_started":1}
I20260806 09:21:51.555181 23850 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88 (DEFAULT_TABLE table=test-table [id=62a607892e3245909ceac01603179b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:51.555272 23645 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88 (DEFAULT_TABLE table=test-table [id=62a607892e3245909ceac01603179b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:51.555262 23460 tablet_service.cc:1467] Processing CreateTablet for tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88 (DEFAULT_TABLE table=test-table [id=62a607892e3245909ceac01603179b09]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:21:51.556675 23850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:51.557020 23460 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:51.557047 23645 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:51.569250 23979 tablet_bootstrap.cc:492] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1: Bootstrap starting.
I20260806 09:21:51.573191 23979 tablet_bootstrap.cc:654] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:51.574422 23979 log.cc:824] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:51.575580 23980 tablet_bootstrap.cc:492] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6: Bootstrap starting.
I20260806 09:21:51.576262 23981 tablet_bootstrap.cc:492] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553: Bootstrap starting.
I20260806 09:21:51.580627 23980 tablet_bootstrap.cc:654] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:51.582216 23980 log.cc:824] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:51.582589 23981 tablet_bootstrap.cc:654] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:51.584086 23981 log.cc:824] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:51.584749 23979 tablet_bootstrap.cc:492] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1: No bootstrap required, opened a new log
I20260806 09:21:51.585161 23979 ts_tablet_manager.cc:1397] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.012s
I20260806 09:21:51.585925 23980 tablet_bootstrap.cc:492] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6: No bootstrap required, opened a new log
I20260806 09:21:51.586233 23980 ts_tablet_manager.cc:1397] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20260806 09:21:51.587339 23981 tablet_bootstrap.cc:492] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553: No bootstrap required, opened a new log
I20260806 09:21:51.587653 23981 ts_tablet_manager.cc:1397] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260806 09:21:51.595492 23981 raft_consensus.cc:348] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 46383 } } peers { permanent_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34781 } } peers { permanent_uuid: "5ca349add8c241d3880f827976ddb553" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35687 } }
I20260806 09:21:51.595944 23981 raft_consensus.cc:374] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:51.596038 23981 raft_consensus.cc:729] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5ca349add8c241d3880f827976ddb553, State: Initialized, Role: FOLLOWER
I20260806 09:21:51.596700 23981 consensus_queue.cc:260] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 46383 } } peers { permanent_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34781 } } peers { permanent_uuid: "5ca349add8c241d3880f827976ddb553" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35687 } }
I20260806 09:21:51.597502 23980 raft_consensus.cc:348] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 46383 } } peers { permanent_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34781 } } peers { permanent_uuid: "5ca349add8c241d3880f827976ddb553" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35687 } }
I20260806 09:21:51.597491 23979 raft_consensus.cc:348] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 46383 } } peers { permanent_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34781 } } peers { permanent_uuid: "5ca349add8c241d3880f827976ddb553" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35687 } }
I20260806 09:21:51.597998 23748 heartbeater.cc:503] Master 127.16.86.190:43015 was elected leader, sending a full tablet report...
I20260806 09:21:51.598074 23979 raft_consensus.cc:374] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:51.598074 23980 raft_consensus.cc:374] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:51.598183 23979 raft_consensus.cc:729] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 52b7a22fad9b49f6b3729a5fe2d45fc1, State: Initialized, Role: FOLLOWER
I20260806 09:21:51.598183 23980 raft_consensus.cc:729] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 254fbe73f1f24ba196bb46becf8ef6b6, State: Initialized, Role: FOLLOWER
I20260806 09:21:51.598754 23981 ts_tablet_manager.cc:1428] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20260806 09:21:51.598978 23979 consensus_queue.cc:260] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 46383 } } peers { permanent_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34781 } } peers { permanent_uuid: "5ca349add8c241d3880f827976ddb553" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35687 } }
I20260806 09:21:51.598979 23980 consensus_queue.cc:260] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 46383 } } peers { permanent_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34781 } } peers { permanent_uuid: "5ca349add8c241d3880f827976ddb553" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35687 } }
I20260806 09:21:51.601641 23979 ts_tablet_manager.cc:1428] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1: Time spent starting tablet: real 0.016s user 0.014s sys 0.003s
I20260806 09:21:51.604020 23954 heartbeater.cc:503] Master 127.16.86.190:43015 was elected leader, sending a full tablet report...
I20260806 09:21:51.604676 23980 ts_tablet_manager.cc:1428] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6: Time spent starting tablet: real 0.018s user 0.015s sys 0.004s
I20260806 09:21:51.605299 23986 raft_consensus.cc:482] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:21:51.605667 23986 raft_consensus.cc:504] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 46383 } } peers { permanent_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34781 } } peers { permanent_uuid: "5ca349add8c241d3880f827976ddb553" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35687 } }
I20260806 09:21:51.610375 23986 leader_election.cc:288] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 52b7a22fad9b49f6b3729a5fe2d45fc1 (127.16.86.131:34781), 5ca349add8c241d3880f827976ddb553 (127.16.86.130:35687)
I20260806 09:21:51.614630 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.609089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:51129 (local address 127.16.86.131:34781)
0806 09:21:51.610132 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:51.610141 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:51.611731 (+ 1590us) server_negotiation.cc:407] Connection header received
0806 09:21:51.612005 (+ 274us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:51.612011 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:51.612082 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:51.612158 (+ 76us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:51.613815 (+ 1657us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:51.613823 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:51.613827 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:51.613855 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:51.613907 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:51.613925 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:51.614080 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:51.614221 (+ 141us) server_negotiation.cc:299] Negotiation successful
0806 09:21:51.614296 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":866}
I20260806 09:21:51.615077 23992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.608625 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:34781 (local address 127.16.86.129:51129)
0806 09:21:51.611600 (+ 2975us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:51.611625 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:51.611843 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:51.612407 (+ 564us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:51.612414 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:51.612442 (+ 28us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:51.613521 (+ 1079us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:21:51.613524 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:51.613547 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:21:51.613602 (+ 55us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:51.614073 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:51.614081 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:51.614089 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 09:21:51.614283 (+ 194us) client_negotiation.cc:240] Negotiation successful
0806 09:21:51.614358 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2809,"thread_start_us":1196,"threads_started":1}
I20260806 09:21:51.615932 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.610946 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:37873 (local address 127.16.86.130:35687)
0806 09:21:51.611268 (+ 322us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:51.611275 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:51.613315 (+ 2040us) server_negotiation.cc:407] Connection header received
0806 09:21:51.613509 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:51.613515 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:51.613587 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:51.613672 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:51.615118 (+ 1446us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:51.615125 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:51.615131 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:51.615157 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:51.615200 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:51.615218 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:51.615323 (+ 105us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:51.615542 (+ 219us) server_negotiation.cc:299] Negotiation successful
0806 09:21:51.615617 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":144}
I20260806 09:21:51.616395 23880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae2940bf8ef45c98de4ffb5c2ac0e88" candidate_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1" is_pre_election: true
I20260806 09:21:51.617048 23880 raft_consensus.cc:2454] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 254fbe73f1f24ba196bb46becf8ef6b6 in term 0.
I20260806 09:21:51.617878 23993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.611454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:35687 (local address 127.16.86.129:37873)
0806 09:21:51.613177 (+ 1723us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:51.613193 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:51.613338 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:51.613894 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:51.613898 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:51.613916 (+ 18us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:21:51.614764 (+ 848us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:21:51.614768 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:21:51.614789 (+ 21us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:21:51.614847 (+ 58us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:21:51.615402 (+ 555us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:21:51.615406 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:21:51.615408 (+ 2us) client_negotiation.cc:769] Sending connection context
0806 09:21:51.615568 (+ 160us) client_negotiation.cc:240] Negotiation successful
0806 09:21:51.615622 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1613,"thread_start_us":1337,"threads_started":1}
I20260806 09:21:51.617910 23395 leader_election.cc:302] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 254fbe73f1f24ba196bb46becf8ef6b6, 52b7a22fad9b49f6b3729a5fe2d45fc1; no voters:
I20260806 09:21:51.618417 23986 raft_consensus.cc:2781] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:21:51.618608 23986 raft_consensus.cc:482] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:21:51.618726 23986 raft_consensus.cc:3037] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:51.619290 23676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae2940bf8ef45c98de4ffb5c2ac0e88" candidate_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ca349add8c241d3880f827976ddb553" is_pre_election: true
I20260806 09:21:51.619829 23676 raft_consensus.cc:2454] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 254fbe73f1f24ba196bb46becf8ef6b6 in term 0.
I20260806 09:21:51.621775 23986 raft_consensus.cc:504] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 46383 } } peers { permanent_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34781 } } peers { permanent_uuid: "5ca349add8c241d3880f827976ddb553" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35687 } }
I20260806 09:21:51.622845 23986 leader_election.cc:288] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [CANDIDATE]: Term 1 election: Requested vote from peers 52b7a22fad9b49f6b3729a5fe2d45fc1 (127.16.86.131:34781), 5ca349add8c241d3880f827976ddb553 (127.16.86.130:35687)
I20260806 09:21:51.623569 23676 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae2940bf8ef45c98de4ffb5c2ac0e88" candidate_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5ca349add8c241d3880f827976ddb553"
I20260806 09:21:51.623654 23880 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1ae2940bf8ef45c98de4ffb5c2ac0e88" candidate_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1"
I20260806 09:21:51.623833 23676 raft_consensus.cc:3037] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:51.623915 23880 raft_consensus.cc:3037] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:51.627842 23880 raft_consensus.cc:2454] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 254fbe73f1f24ba196bb46becf8ef6b6 in term 1.
I20260806 09:21:51.627841 23676 raft_consensus.cc:2454] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 254fbe73f1f24ba196bb46becf8ef6b6 in term 1.
I20260806 09:21:51.628454 23395 leader_election.cc:302] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 254fbe73f1f24ba196bb46becf8ef6b6, 52b7a22fad9b49f6b3729a5fe2d45fc1; no voters:
I20260806 09:21:51.628924 23986 raft_consensus.cc:2781] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:51.629245 23986 raft_consensus.cc:686] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 1 LEADER]: Becoming Leader. State: Replica: 254fbe73f1f24ba196bb46becf8ef6b6, State: Running, Role: LEADER
I20260806 09:21:51.629987 23986 consensus_queue.cc:237] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 46383 } } peers { permanent_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34781 } } peers { permanent_uuid: "5ca349add8c241d3880f827976ddb553" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35687 } }
I20260806 09:21:51.635887 23282 catalog_manager.cc:5495] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 reported cstate change: term changed from 0 to 1, leader changed from <none> to 254fbe73f1f24ba196bb46becf8ef6b6 (127.16.86.129). New cstate: current_term: 1 leader_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "254fbe73f1f24ba196bb46becf8ef6b6" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 46383 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34781 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5ca349add8c241d3880f827976ddb553" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35687 } health_report { overall_health: UNKNOWN } } }
I20260806 09:21:51.656826 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.654365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51500 (local address 127.16.86.129:46383)
0806 09:21:51.654579 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:51.654584 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:51.654607 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:21:51.654890 (+ 283us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:51.654894 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:51.655071 (+ 177us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:51.655182 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:51.656225 (+ 1043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:51.656229 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:51.656233 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:51.656249 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:51.656288 (+ 39us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:51.656301 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:51.656419 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:51.656530 (+ 111us) server_negotiation.cc:299] Negotiation successful
0806 09:21:51.656591 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 09:21:51.660454 23753 tablet.cc:2369] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:21:51.664803 23556 tablet.cc:2369] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:21:51.668359 23676 raft_consensus.cc:1261] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553 [term 1 FOLLOWER]: Refusing update from remote peer 254fbe73f1f24ba196bb46becf8ef6b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:51.668664 23880 raft_consensus.cc:1261] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1 [term 1 FOLLOWER]: Refusing update from remote peer 254fbe73f1f24ba196bb46becf8ef6b6: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:21:51.669807 23989 consensus_queue.cc:1035] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5ca349add8c241d3880f827976ddb553" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 35687 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:51.670290 23986 consensus_queue.cc:1035] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "52b7a22fad9b49f6b3729a5fe2d45fc1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34781 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:21:51.684583 23998 mvcc.cc:205] Tried to move back new op lower bound from 7315489225378721792 to 7315489225241780224. Current Snapshot: MvccSnapshot[applied={T|T < 7315489225378721792}]
I20260806 09:21:51.701771 23275 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:51.702239 23275 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35322:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:21:51.708119 23275 catalog_manager.cc:5782] T 00000000000000000000000000000000 P be0317b2566646ce8bc0956993ccc465: Sending DeleteTablet for 3 replicas of tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88
I20260806 09:21:51.709977 23850 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:51 UTC) from {username='slave'} at 127.0.0.1:50630
I20260806 09:21:51.709997 23460 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:51 UTC) from {username='slave'} at 127.0.0.1:51492
I20260806 09:21:51.711171 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 23371
I20260806 09:21:51.711730 23645 tablet_service.cc:1514] Processing DeleteTablet for tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:21:51 UTC) from {username='slave'} at 127.0.0.1:33632
I20260806 09:21:51.721776 24013 tablet_replica.cc:330] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553: stopping tablet replica
I20260806 09:21:51.721868 24010 tablet_replica.cc:330] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6: stopping tablet replica
I20260806 09:21:51.722565 24010 raft_consensus.cc:2229] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:21:51.722780 24013 raft_consensus.cc:2229] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:51.723464 24010 raft_consensus.cc:2258] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:51.724516 24011 tablet_replica.cc:330] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1: stopping tablet replica
I20260806 09:21:51.724584 24013 raft_consensus.cc:2258] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:51.725127 24011 raft_consensus.cc:2229] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:21:51.725592 23972 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.712222 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51506 (local address 127.16.86.129:46383)
0806 09:21:51.712461 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:51.712473 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:51.720248 (+ 7775us) server_negotiation.cc:407] Connection header received
0806 09:21:51.721909 (+ 1661us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:51.721918 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:51.722009 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:51.722114 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:51.724378 (+ 2264us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:51.724385 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:51.724390 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:51.724425 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:51.724482 (+ 57us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:51.724503 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:51.724664 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:51.724991 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 09:21:51.725082 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20260806 09:21:51.725826 24011 raft_consensus.cc:2258] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:21:51.727334 23546 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:51.730150 24011 ts_tablet_manager.cc:1905] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:51.731285 24013 ts_tablet_manager.cc:1905] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:51.740120 24013 ts_tablet_manager.cc:1918] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:21:51.740363 24013 log.cc:1196] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/1ae2940bf8ef45c98de4ffb5c2ac0e88
I20260806 09:21:51.741030 24013 ts_tablet_manager.cc:1939] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 5ca349add8c241d3880f827976ddb553: Deleting consensus metadata
I20260806 09:21:51.741398 24011 ts_tablet_manager.cc:1918] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:21:51.741632 24011 log.cc:1196] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/1ae2940bf8ef45c98de4ffb5c2ac0e88
I20260806 09:21:51.742173 24011 ts_tablet_manager.cc:1939] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 52b7a22fad9b49f6b3729a5fe2d45fc1: Deleting consensus metadata
I20260806 09:21:51.743516 23259 catalog_manager.cc:4841] TS 5ca349add8c241d3880f827976ddb553 (127.16.86.130:35687): tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88 (table test-table [id=62a607892e3245909ceac01603179b09]) successfully deleted
I20260806 09:21:51.744727 23257 catalog_manager.cc:4841] TS 52b7a22fad9b49f6b3729a5fe2d45fc1 (127.16.86.131:34781): tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88 (table test-table [id=62a607892e3245909ceac01603179b09]) successfully deleted
I20260806 09:21:51.727610 24010 ts_tablet_manager.cc:1905] T 1ae2940bf8ef45c98de4ffb5c2ac0e88 P 254fbe73f1f24ba196bb46becf8ef6b6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:21:51.840416 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 23371
W20260806 09:21:51.866844 23259 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.86.129:46383 (error 108)
I20260806 09:21:51.867878 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 23562
I20260806 09:21:51.869155 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.868465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:46383 (local address 127.0.0.1:51518)
0806 09:21:51.868752 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:51.868966 (+ 214us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260806 09:21:51.869481 23259 catalog_manager.cc:4584] TS 254fbe73f1f24ba196bb46becf8ef6b6 (127.16.86.129:46383): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 1ae2940bf8ef45c98de4ffb5c2ac0e88: Network error: Client connection negotiation failed: client connection to 127.16.86.129:46383: connect: Connection refused (error 111)
I20260806 09:21:51.871790 23978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.868977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33634 (local address 127.16.86.130:35687)
0806 09:21:51.869171 (+ 194us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:51.869176 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:51.869198 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:21:51.869486 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:51.869495 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:51.869652 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:51.869804 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:51.870980 (+ 1176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:51.870986 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:51.870989 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:51.871054 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:51.871097 (+ 43us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:51.871111 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:51.871221 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:51.871388 (+ 167us) server_negotiation.cc:299] Negotiation successful
0806 09:21:51.871447 (+ 59us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:21:51.872663 23738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:51.924190 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.923423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:46383 (local address 127.0.0.1:51534)
0806 09:21:51.923774 (+ 351us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:51.923957 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260806 09:21:51.925741 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.925265 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:46383 (local address 127.0.0.1:51546)
0806 09:21:51.925528 (+ 263us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:51.925630 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20260806 09:21:51.977029 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 23562
I20260806 09:21:51.986945 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.986411 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:46383 (local address 127.0.0.1:51550)
0806 09:21:51.986645 (+ 234us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:51.986775 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 09:21:51.988441 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:51.987925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:46383 (local address 127.0.0.1:51552)
0806 09:21:51.988151 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:51.988252 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20260806 09:21:52.006955 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 23755
I20260806 09:21:52.010846 23976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:52.008238 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50632 (local address 127.16.86.131:34781)
0806 09:21:52.008526 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:52.008531 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:52.008562 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:21:52.008666 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:52.008672 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:52.008790 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:52.008920 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:52.010016 (+ 1096us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:52.010021 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:52.010025 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:52.010043 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:52.010095 (+ 52us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:52.010110 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:52.010261 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:52.010319 (+ 58us) server_negotiation.cc:299] Negotiation successful
0806 09:21:52.010382 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20260806 09:21:52.011922 23945 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:52.069362 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:52.068574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:46383 (local address 127.0.0.1:51556)
0806 09:21:52.068982 (+ 408us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:52.069142 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":151}
I20260806 09:21:52.070777 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:52.070309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:46383 (local address 127.0.0.1:51566)
0806 09:21:52.070574 (+ 265us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:52.070647 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260806 09:21:52.107653 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 23755
I20260806 09:21:52.134157 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 23232
I20260806 09:21:52.135777 23320 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:52.250767 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 23232
I20260806 09:21:52.250801 23974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:52.250061 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:46383 (local address 127.0.0.1:51572)
0806 09:21:52.250354 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:52.250530 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:46383: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
2026-08-06T09:21:52Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (3041 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20260806 09:21:52.310410 16730 mini_oidc.cc:152] Starting JWKS server
I20260806 09:21:52.314616 16730 webserver.cc:460] Webserver started at https://127.0.0.1:38671/ using document root <none> and password file <none>
I20260806 09:21:52.315181 16730 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 09:21:52.316133 16730 webserver.cc:460] Webserver started at http://127.0.0.1:42539/ using document root <none> and password file <none>
2026-08-06T09:21:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:52Z Disabled control of system clock
I20260806 09:21:52.364082 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:36415
--webserver_interface=127.16.86.190
--webserver_port=0
--builtin_ntp_servers=127.16.86.148:43823
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:36415
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38671/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/testchainca.cert with env {}
W20260806 09:21:52.758077 24027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:52.758477 24027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:52.758666 24027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:52.766376 24027 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:52.766547 24027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:52.766649 24027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:52.766757 24027 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:52.766849 24027 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 09:21:52.766942 24027 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 09:21:52.767042 24027 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:38671/jwks.json
I20260806 09:21:52.777740 24027 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:43823
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:36415
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:36415
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:38671/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:52.779217 24027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:52.780593 24027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:52.794279 24036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:52.794353 24034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:52.794802 24033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:52.796060 24027 server_base.cc:1034] running on GCE node
I20260806 09:21:52.797202 24027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:52.799203 24027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:52.800462 24027 hybrid_clock.cc:648] HybridClock initialized: now 1786008112800408 us; error 50 us; skew 500 ppm
I20260806 09:21:52.800909 24027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:21:52.808312 24027 webserver.cc:460] Webserver started at http://127.16.86.190:36565/ using document root <none> and password file <none>
I20260806 09:21:52.809260 24027 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:52.809422 24027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:52.809821 24027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:52.813629 24027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "3b249770bff24ff28aaadc15d0c8d683"
format_stamp: "Formatted at 2026-08-06 09:21:52 on dist-test-slave-h3ft"
server_key: "dfcbea196554ccbb9121bcf92d947347"
server_key_iv: "0af6a0fa8d5e232cc1ff40df16276010"
server_key_version: "encryptionkey@0"
I20260806 09:21:52.814617 24027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "3b249770bff24ff28aaadc15d0c8d683"
format_stamp: "Formatted at 2026-08-06 09:21:52 on dist-test-slave-h3ft"
server_key: "dfcbea196554ccbb9121bcf92d947347"
server_key_iv: "0af6a0fa8d5e232cc1ff40df16276010"
server_key_version: "encryptionkey@0"
I20260806 09:21:52.821554 24027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:21:52.826228 24049 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:52.827961 24027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:21:52.828167 24027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "3b249770bff24ff28aaadc15d0c8d683"
format_stamp: "Formatted at 2026-08-06 09:21:52 on dist-test-slave-h3ft"
server_key: "dfcbea196554ccbb9121bcf92d947347"
server_key_iv: "0af6a0fa8d5e232cc1ff40df16276010"
server_key_version: "encryptionkey@0"
I20260806 09:21:52.828392 24027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:52.861059 24027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:52.862097 24027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:52.862447 24027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:52.883769 24027 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:36415
I20260806 09:21:52.883834 24120 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:36415 every 8 connection(s)
I20260806 09:21:52.885304 24027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:52.887015 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 24027
I20260806 09:21:52.887288 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:52.888226 16730 external_mini_cluster.cc:1428] Setting key f5e1c0334f7ee691bb0b96d307be596d
I20260806 09:21:52.891646 24121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:52.904264 24121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683: Bootstrap starting.
I20260806 09:21:52.904409 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:52.890658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53818 (local address 127.16.86.190:36415)
0806 09:21:52.892118 (+ 1460us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:52.892134 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:52.892191 (+ 57us) server_negotiation.cc:407] Connection header received
0806 09:21:52.894128 (+ 1937us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:52.894153 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:52.894762 (+ 609us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:52.895315 (+ 553us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:52.896000 (+ 685us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:52.897619 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:52.899163 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:52.899895 (+ 732us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:52.900143 (+ 248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:52.901581 (+ 1438us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:52.901590 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:52.901607 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:52.902295 (+ 688us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:52.902380 (+ 85us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:52.902404 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:52.902556 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:52.902814 (+ 258us) server_negotiation.cc:299] Negotiation successful
0806 09:21:52.903276 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":530,"threads_started":1}
I20260806 09:21:52.908885 24121 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:52.909427 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:21:52.910226 24121 log.cc:824] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:52.913870 24121 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683: No bootstrap required, opened a new log
I20260806 09:21:52.922283 24121 raft_consensus.cc:348] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b249770bff24ff28aaadc15d0c8d683" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36415 } }
I20260806 09:21:52.922751 24121 raft_consensus.cc:374] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:52.922873 24121 raft_consensus.cc:729] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b249770bff24ff28aaadc15d0c8d683, State: Initialized, Role: FOLLOWER
I20260806 09:21:52.923636 24121 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b249770bff24ff28aaadc15d0c8d683" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36415 } }
I20260806 09:21:52.923913 24121 raft_consensus.cc:388] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:52.924073 24121 raft_consensus.cc:482] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:52.924280 24121 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:52.927156 24121 raft_consensus.cc:504] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b249770bff24ff28aaadc15d0c8d683" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36415 } }
I20260806 09:21:52.927816 24121 leader_election.cc:302] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3b249770bff24ff28aaadc15d0c8d683; no voters:
I20260806 09:21:52.928530 24121 leader_election.cc:288] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:52.928845 24128 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:52.929987 24128 raft_consensus.cc:686] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [term 1 LEADER]: Becoming Leader. State: Replica: 3b249770bff24ff28aaadc15d0c8d683, State: Running, Role: LEADER
I20260806 09:21:52.931182 24128 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b249770bff24ff28aaadc15d0c8d683" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36415 } }
I20260806 09:21:52.931732 24121 sys_catalog.cc:564] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:52.934804 24129 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3b249770bff24ff28aaadc15d0c8d683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b249770bff24ff28aaadc15d0c8d683" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36415 } } }
I20260806 09:21:52.935520 24129 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:52.935338 24132 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3b249770bff24ff28aaadc15d0c8d683. Latest consensus state: current_term: 1 leader_uuid: "3b249770bff24ff28aaadc15d0c8d683" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3b249770bff24ff28aaadc15d0c8d683" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36415 } } }
I20260806 09:21:52.935920 24132 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:52.939445 24135 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:52.946806 24135 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:52.961341 24135 catalog_manager.cc:1347] Generated new cluster ID: 9fabbcd54c4c42b6a93f04129869d55e
I20260806 09:21:52.961558 24135 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:53.000308 24135 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:53.002161 24135 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:53.011365 24135 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 3b249770bff24ff28aaadc15d0c8d683: Generated new TSK 0
I20260806 09:21:53.012578 24135 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 09:21:53.069242 24162 webserver.cc:533] Webserver: sslize failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
E20260806 09:21:53.069583 24162 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 09:21:53.071070 24125 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:53.047937 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.16.86.190:36415)
0806 09:21:53.048248 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:53.048258 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:53.048745 (+ 487us) server_negotiation.cc:407] Connection header received
0806 09:21:53.049233 (+ 488us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:53.049238 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:53.049333 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:53.049504 (+ 171us) server_negotiation.cc:226] Negotiated authn=JWT
0806 09:21:53.050155 (+ 651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:53.051077 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:53.052648 (+ 1571us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:53.053192 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:53.053366 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:53.054023 (+ 657us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 09:21:53.070144 (+ 16121us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 09:21:53.070638 (+ 494us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53826: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"server-negotiator.queue_time_us":99}
W20260806 09:21:53.071228 24125 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53826: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
W20260806 09:21:53.073211 24160 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:21:53.047816 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:36415 (local address 127.0.0.1:53826)
0806 09:21:53.048598 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:53.048625 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:53.048876 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:53.049549 (+ 673us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:53.049555 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:53.049571 (+ 16us) client_negotiation.cc:189] Negotiated authn=JWT
0806 09:21:53.049978 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:53.049989 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:53.051224 (+ 1235us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:53.051230 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:53.052464 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:53.052476 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:53.053412 (+ 936us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:53.053419 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:53.053595 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:53.053673 (+ 78us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 09:21:53.070554 (+ 16881us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
0806 09:21:53.071030 (+ 476us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.86.190:36415: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":403,"threads_started":1}
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:597: Failure
Failed
Bad status: Runtime error: Could not connect to the cluster: Client connection negotiation failed: client connection to 127.16.86.190:36415: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:38671/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: certificate has expired
Google Test trace:
/data0/somelongdirectorytoavoidrpathissues/src/kudu/src/kudu/integration-tests/security-itest.cc:593: Valid JWT
I20260806 09:21:53.075460 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 24027
I20260806 09:21:53.076977 24114 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:53.180845 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 24027
2026-08-06T09:21:53Z chronyd exiting
I20260806 09:21:53.241832 16730 test_util.cc:175] -----------------------------------------------
I20260806 09:21:53.242105 16730 test_util.cc:176] Had fatal failures, leaving test files at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1786008061536593-16730-0
[ FAILED ] SecurityITest.TestJwtMiniCluster (938 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20260806 09:21:53.246773 16730 mini_oidc.cc:152] Starting JWKS server
I20260806 09:21:53.249284 16730 webserver.cc:460] Webserver started at https://127.0.0.1:37905/ using document root <none> and password file <none>
I20260806 09:21:53.249577 16730 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 09:21:53.250389 16730 webserver.cc:460] Webserver started at http://127.0.0.1:43555/ using document root <none> and password file <none>
2026-08-06T09:21:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:53Z Disabled control of system clock
I20260806 09:21:53.298007 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41275
--webserver_interface=127.16.86.190
--webserver_port=0
--builtin_ntp_servers=127.16.86.148:44197
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:41275
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37905/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/testchainca.cert with env {}
W20260806 09:21:53.649230 24171 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:53.649657 24171 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:53.649809 24171 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:53.657486 24171 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:53.657670 24171 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:53.657774 24171 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:53.657903 24171 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:53.657996 24171 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 09:21:53.658113 24171 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 09:21:53.658196 24171 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:37905/jwks.json
I20260806 09:21:53.668560 24171 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44197
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:41275
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41275
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:37905/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:53.670080 24171 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:53.671509 24171 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:53.684762 24178 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:53.685181 24184 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:53.684891 24180 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:53.686431 24171 server_base.cc:1034] running on GCE node
I20260806 09:21:53.687597 24171 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:53.689647 24171 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:53.690917 24171 hybrid_clock.cc:648] HybridClock initialized: now 1786008113690838 us; error 72 us; skew 500 ppm
I20260806 09:21:53.691433 24171 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:21:53.694428 24171 webserver.cc:460] Webserver started at http://127.16.86.190:35039/ using document root <none> and password file <none>
I20260806 09:21:53.695410 24171 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:53.695560 24171 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:53.695977 24171 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:53.699644 24171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "b0d7978ba5684820b468795a9248b3bf"
format_stamp: "Formatted at 2026-08-06 09:21:53 on dist-test-slave-h3ft"
server_key: "345eaebbc70c2d71edde95b474798b4c"
server_key_iv: "7e8df5a4b3b31b28080700de225aa84c"
server_key_version: "encryptionkey@0"
I20260806 09:21:53.700668 24171 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "b0d7978ba5684820b468795a9248b3bf"
format_stamp: "Formatted at 2026-08-06 09:21:53 on dist-test-slave-h3ft"
server_key: "345eaebbc70c2d71edde95b474798b4c"
server_key_iv: "7e8df5a4b3b31b28080700de225aa84c"
server_key_version: "encryptionkey@0"
I20260806 09:21:53.707423 24171 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 09:21:53.712222 24192 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:53.714126 24171 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:21:53.714403 24171 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "b0d7978ba5684820b468795a9248b3bf"
format_stamp: "Formatted at 2026-08-06 09:21:53 on dist-test-slave-h3ft"
server_key: "345eaebbc70c2d71edde95b474798b4c"
server_key_iv: "7e8df5a4b3b31b28080700de225aa84c"
server_key_version: "encryptionkey@0"
I20260806 09:21:53.714704 24171 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:53.730031 24171 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:53.731163 24171 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:53.731537 24171 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:53.754072 24171 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:41275
I20260806 09:21:53.754104 24266 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:41275 every 8 connection(s)
I20260806 09:21:53.755964 24171 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:53.760512 24267 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:53.760928 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 24171
I20260806 09:21:53.761188 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:53.761962 16730 external_mini_cluster.cc:1428] Setting key 1e748491ed26075bc7f4bf9e5e53a166
I20260806 09:21:53.775449 24267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf: Bootstrap starting.
I20260806 09:21:53.778705 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:53.764402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35332 (local address 127.16.86.190:41275)
0806 09:21:53.765999 (+ 1597us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:53.766039 (+ 40us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:53.766112 (+ 73us) server_negotiation.cc:407] Connection header received
0806 09:21:53.768356 (+ 2244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:53.768385 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:53.769009 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:53.769497 (+ 488us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:53.770224 (+ 727us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:53.771735 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:53.773177 (+ 1442us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:53.773946 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:53.774172 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:53.775816 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:53.775830 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:53.775856 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:53.776558 (+ 702us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:53.776657 (+ 99us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:53.776682 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:53.776827 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:53.777120 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 09:21:53.777507 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":878,"thread_start_us":578,"threads_started":1}
I20260806 09:21:53.779433 24267 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:53.780779 24267 log.cc:824] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:53.783849 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:21:53.784337 24267 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf: No bootstrap required, opened a new log
I20260806 09:21:53.792855 24267 raft_consensus.cc:348] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d7978ba5684820b468795a9248b3bf" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41275 } }
I20260806 09:21:53.793368 24267 raft_consensus.cc:374] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:53.793501 24267 raft_consensus.cc:729] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0d7978ba5684820b468795a9248b3bf, State: Initialized, Role: FOLLOWER
I20260806 09:21:53.794219 24267 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d7978ba5684820b468795a9248b3bf" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41275 } }
I20260806 09:21:53.794457 24267 raft_consensus.cc:388] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:53.794565 24267 raft_consensus.cc:482] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:53.794714 24267 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:53.796939 24267 raft_consensus.cc:504] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d7978ba5684820b468795a9248b3bf" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41275 } }
I20260806 09:21:53.797840 24267 leader_election.cc:302] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b0d7978ba5684820b468795a9248b3bf; no voters:
I20260806 09:21:53.798619 24267 leader_election.cc:288] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:53.798885 24272 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:53.799947 24272 raft_consensus.cc:686] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [term 1 LEADER]: Becoming Leader. State: Replica: b0d7978ba5684820b468795a9248b3bf, State: Running, Role: LEADER
I20260806 09:21:53.800627 24272 consensus_queue.cc:237] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d7978ba5684820b468795a9248b3bf" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41275 } }
I20260806 09:21:53.801368 24267 sys_catalog.cc:564] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:53.804512 24273 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b0d7978ba5684820b468795a9248b3bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d7978ba5684820b468795a9248b3bf" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41275 } } }
I20260806 09:21:53.805006 24273 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:53.806861 24276 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [sys.catalog]: SysCatalogTable state changed. Reason: New leader b0d7978ba5684820b468795a9248b3bf. Latest consensus state: current_term: 1 leader_uuid: "b0d7978ba5684820b468795a9248b3bf" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b0d7978ba5684820b468795a9248b3bf" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41275 } } }
I20260806 09:21:53.807230 24276 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:53.808039 24286 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:53.815442 24286 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:53.828349 24286 catalog_manager.cc:1347] Generated new cluster ID: bb106013969c479b8c134fc4100994df
I20260806 09:21:53.828505 24286 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:53.843940 24286 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:53.845333 24286 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:53.854278 24286 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b0d7978ba5684820b468795a9248b3bf: Generated new TSK 0
I20260806 09:21:53.855746 24286 catalog_manager.cc:1514] Initializing in-progress tserver states...
E20260806 09:21:53.900699 24311 webserver.cc:533] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20260806 09:21:53.900887 24311 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 09:21:53.902518 24270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:53.878694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35348 (local address 127.16.86.190:41275)
0806 09:21:53.878959 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:53.878968 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:53.879618 (+ 650us) server_negotiation.cc:407] Connection header received
0806 09:21:53.879908 (+ 290us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:53.879915 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:53.880049 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:53.880300 (+ 251us) server_negotiation.cc:226] Negotiated authn=JWT
0806 09:21:53.880852 (+ 552us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:53.882067 (+ 1215us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:53.883714 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:53.884486 (+ 772us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:53.884644 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:53.885252 (+ 608us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 09:21:53.901592 (+ 16340us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 09:21:53.902110 (+ 518us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":76}
W20260806 09:21:53.902817 24270 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35348: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
W20260806 09:21:53.903359 24310 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:21:53.878676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41275 (local address 127.0.0.1:35348)
0806 09:21:53.879491 (+ 815us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:53.879511 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:53.879661 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:53.880321 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:53.880325 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:53.880337 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0806 09:21:53.880650 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:53.880659 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:53.882273 (+ 1614us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:53.882278 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:53.883554 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:53.883567 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:53.884702 (+ 1135us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:53.884707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:53.884868 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:53.884908 (+ 40us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 09:21:53.902138 (+ 17230us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
0806 09:21:53.902458 (+ 320us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.86.190:41275: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:37905/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":651,"thread_start_us":469,"threads_started":1}
I20260806 09:21:53.905210 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 24171
I20260806 09:21:53.907240 24260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:54.017127 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 24171
2026-08-06T09:21:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (839 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20260806 09:21:54.086010 16730 mini_oidc.cc:152] Starting JWKS server
I20260806 09:21:54.088433 16730 webserver.cc:460] Webserver started at https://127.0.0.1:43891/ using document root <none> and password file <none>
I20260806 09:21:54.088761 16730 mini_oidc.cc:189] Starting OIDC Discovery server
I20260806 09:21:54.089550 16730 webserver.cc:460] Webserver started at http://127.0.0.1:38849/ using document root <none> and password file <none>
2026-08-06T09:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:54Z Disabled control of system clock
I20260806 09:21:54.137914 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:43243
--webserver_interface=127.16.86.190
--webserver_port=0
--builtin_ntp_servers=127.16.86.148:40661
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:43243
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43891/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 09:21:54.483063 24321 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:54.483439 24321 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:54.483618 24321 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:54.490989 24321 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:54.491178 24321 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:54.491272 24321 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:54.491384 24321 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:54.491475 24321 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 09:21:54.491569 24321 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 09:21:54.491649 24321 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43891/jwks.json
I20260806 09:21:54.500959 24321 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:40661
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:43243
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:43243
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43891/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:54.502385 24321 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:54.504240 24321 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:54.517254 24330 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:54.517321 24331 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:54.517856 24334 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:54.518908 24321 server_base.cc:1034] running on GCE node
I20260806 09:21:54.519902 24321 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:54.521611 24321 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:54.522831 24321 hybrid_clock.cc:648] HybridClock initialized: now 1786008114522753 us; error 89 us; skew 500 ppm
I20260806 09:21:54.523239 24321 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:21:54.525779 24321 webserver.cc:460] Webserver started at http://127.16.86.190:45595/ using document root <none> and password file <none>
I20260806 09:21:54.526614 24321 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:54.526758 24321 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:54.527227 24321 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:54.530167 24321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "41d9c6fcbc644b909c62946bc49680af"
format_stamp: "Formatted at 2026-08-06 09:21:54 on dist-test-slave-h3ft"
server_key: "cf4894532edd549bb4a0a6dae2bfc61f"
server_key_iv: "903dbd65fd05e5e3a72927000733cfbe"
server_key_version: "encryptionkey@0"
I20260806 09:21:54.531098 24321 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "41d9c6fcbc644b909c62946bc49680af"
format_stamp: "Formatted at 2026-08-06 09:21:54 on dist-test-slave-h3ft"
server_key: "cf4894532edd549bb4a0a6dae2bfc61f"
server_key_iv: "903dbd65fd05e5e3a72927000733cfbe"
server_key_version: "encryptionkey@0"
I20260806 09:21:54.537026 24321 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 09:21:54.541277 24342 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:54.542903 24321 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:21:54.543191 24321 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "41d9c6fcbc644b909c62946bc49680af"
format_stamp: "Formatted at 2026-08-06 09:21:54 on dist-test-slave-h3ft"
server_key: "cf4894532edd549bb4a0a6dae2bfc61f"
server_key_iv: "903dbd65fd05e5e3a72927000733cfbe"
server_key_version: "encryptionkey@0"
I20260806 09:21:54.543460 24321 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:54.574432 24321 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:54.575623 24321 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:54.575987 24321 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:54.598837 24321 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:43243
I20260806 09:21:54.598899 24420 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:43243 every 8 connection(s)
I20260806 09:21:54.600857 24321 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:54.607693 24421 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:54.609961 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 24321
I20260806 09:21:54.610292 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:54.611419 16730 external_mini_cluster.cc:1428] Setting key e562be7904f77eb19e8a8cf0c895ec35
I20260806 09:21:54.631968 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:54.613975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.16.86.190:43243)
0806 09:21:54.615547 (+ 1572us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:54.615569 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:54.615637 (+ 68us) server_negotiation.cc:407] Connection header received
0806 09:21:54.619950 (+ 4313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:54.619989 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:54.620789 (+ 800us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:54.621348 (+ 559us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:54.623084 (+ 1736us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:54.624434 (+ 1350us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:54.625906 (+ 1472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:54.626683 (+ 777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:54.626900 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:54.629015 (+ 2115us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:54.629031 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:54.629050 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:54.629935 (+ 885us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:54.630088 (+ 153us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:54.630123 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:54.630264 (+ 141us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:54.630503 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 09:21:54.630853 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":449,"threads_started":1}
I20260806 09:21:54.632725 24421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af: Bootstrap starting.
I20260806 09:21:54.637156 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:21:54.638384 24421 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:54.640048 24421 log.cc:824] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:54.647351 24421 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af: No bootstrap required, opened a new log
I20260806 09:21:54.661635 24421 raft_consensus.cc:348] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d9c6fcbc644b909c62946bc49680af" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43243 } }
I20260806 09:21:54.662395 24421 raft_consensus.cc:374] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:54.662534 24421 raft_consensus.cc:729] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 41d9c6fcbc644b909c62946bc49680af, State: Initialized, Role: FOLLOWER
I20260806 09:21:54.663479 24421 consensus_queue.cc:260] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d9c6fcbc644b909c62946bc49680af" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43243 } }
I20260806 09:21:54.663846 24421 raft_consensus.cc:388] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:54.664026 24421 raft_consensus.cc:482] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:54.664239 24421 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:54.668361 24421 raft_consensus.cc:504] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d9c6fcbc644b909c62946bc49680af" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43243 } }
I20260806 09:21:54.669119 24421 leader_election.cc:302] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 41d9c6fcbc644b909c62946bc49680af; no voters:
I20260806 09:21:54.669960 24421 leader_election.cc:288] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:54.670222 24426 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:54.671285 24426 raft_consensus.cc:686] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [term 1 LEADER]: Becoming Leader. State: Replica: 41d9c6fcbc644b909c62946bc49680af, State: Running, Role: LEADER
I20260806 09:21:54.672062 24426 consensus_queue.cc:237] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d9c6fcbc644b909c62946bc49680af" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43243 } }
I20260806 09:21:54.673190 24421 sys_catalog.cc:564] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:54.676293 24428 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "41d9c6fcbc644b909c62946bc49680af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d9c6fcbc644b909c62946bc49680af" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43243 } } }
I20260806 09:21:54.676800 24428 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:54.677644 24429 sys_catalog.cc:455] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [sys.catalog]: SysCatalogTable state changed. Reason: New leader 41d9c6fcbc644b909c62946bc49680af. Latest consensus state: current_term: 1 leader_uuid: "41d9c6fcbc644b909c62946bc49680af" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "41d9c6fcbc644b909c62946bc49680af" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43243 } } }
I20260806 09:21:54.678078 24429 sys_catalog.cc:458] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:54.677963 24441 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:54.685389 24441 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:54.697774 24441 catalog_manager.cc:1347] Generated new cluster ID: bb9c3aa2eb754164a26e38f7c1240e30
I20260806 09:21:54.698002 24441 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:54.731843 24441 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:54.733628 24441 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:54.742597 24441 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 41d9c6fcbc644b909c62946bc49680af: Generated new TSK 0
I20260806 09:21:54.743840 24441 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:21:54.772406 24424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:54.762631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60282 (local address 127.16.86.190:43243)
0806 09:21:54.762933 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:54.762945 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:54.763670 (+ 725us) server_negotiation.cc:407] Connection header received
0806 09:21:54.764058 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:54.764063 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:54.764158 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:54.764359 (+ 201us) server_negotiation.cc:226] Negotiated authn=JWT
0806 09:21:54.765009 (+ 650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:54.765894 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:54.767515 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:54.768086 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:54.768249 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:54.768912 (+ 663us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 09:21:54.771640 (+ 2728us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43891/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 09:21:54.772101 (+ 461us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:60282: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43891/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"server-negotiator.queue_time_us":100}
W20260806 09:21:54.772620 24424 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:60282: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43891/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
W20260806 09:21:54.775738 24457 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:21:54.762617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43243 (local address 127.0.0.1:60282)
0806 09:21:54.763536 (+ 919us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:54.763557 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:54.763775 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:54.764379 (+ 604us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:54.764385 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:54.764398 (+ 13us) client_negotiation.cc:189] Negotiated authn=JWT
0806 09:21:54.764822 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:54.764834 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:54.766066 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:54.766076 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:54.767346 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:54.767359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:54.768335 (+ 976us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:54.768345 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:54.768511 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:54.768555 (+ 44us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 09:21:54.771900 (+ 3345us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43891/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
0806 09:21:54.775315 (+ 3415us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.86.190:43243: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43891/jwks.json': curl error: Problem with the SSL CA cert (path? access rights?): error setting certificate verify locations:
CAfile: /etc/pki/tls/certs/ca-bundle.crt
CApath: none
Metrics: {"client-negotiator.queue_time_us":751,"mutex_wait_us":3065,"thread_start_us":578,"threads_started":1}
I20260806 09:21:54.777073 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 24321
I20260806 09:21:54.778652 24413 generic_service.cc:196] Checking for leaks (request via RPC)
E20260806 09:21:54.787961 24458 webserver.cc:533] Webserver: sslize failed:
E20260806 09:21:54.788111 24458 webserver.cc:533] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20260806 09:21:54.894388 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 24321
2026-08-06T09:21:54Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (883 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2026-08-06T09:21:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:54Z Disabled control of system clock
I20260806 09:21:55.013720 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:43045
--webserver_interface=127.16.86.190
--webserver_port=0
--builtin_ntp_servers=127.16.86.148:44639
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:43045
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/testchainca.cert with env {}
W20260806 09:21:55.370831 24463 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:21:55.371183 24463 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:21:55.371330 24463 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:21:55.378659 24463 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:21:55.378787 24463 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:21:55.378827 24463 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:21:55.378954 24463 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:21:55.379060 24463 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20260806 09:21:55.379173 24463 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20260806 09:21:55.379251 24463 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20260806 09:21:55.389269 24463 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44639
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:43045
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:43045
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:21:55.390600 24463 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:21:55.391956 24463 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:21:55.405433 24477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:55.405627 24479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:21:55.405906 24476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:21:55.406972 24463 server_base.cc:1034] running on GCE node
I20260806 09:21:55.407939 24463 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:21:55.409653 24463 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:21:55.410992 24463 hybrid_clock.cc:648] HybridClock initialized: now 1786008115410912 us; error 82 us; skew 500 ppm
I20260806 09:21:55.411533 24463 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:21:55.414341 24463 webserver.cc:460] Webserver started at http://127.16.86.190:45133/ using document root <none> and password file <none>
I20260806 09:21:55.415205 24463 fs_manager.cc:359] Metadata directory not provided
I20260806 09:21:55.415336 24463 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:21:55.415688 24463 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:21:55.418675 24463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "4a209a58c83b407aad79876883e18244"
format_stamp: "Formatted at 2026-08-06 09:21:55 on dist-test-slave-h3ft"
server_key: "1a0227570c9f399edb2e5f75082d10fc"
server_key_iv: "b6104d1b5abbdab6c108007cbcadf5d1"
server_key_version: "encryptionkey@0"
I20260806 09:21:55.419579 24463 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "4a209a58c83b407aad79876883e18244"
format_stamp: "Formatted at 2026-08-06 09:21:55 on dist-test-slave-h3ft"
server_key: "1a0227570c9f399edb2e5f75082d10fc"
server_key_iv: "b6104d1b5abbdab6c108007cbcadf5d1"
server_key_version: "encryptionkey@0"
I20260806 09:21:55.425096 24463 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:21:55.429028 24489 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:21:55.430670 24463 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260806 09:21:55.430830 24463 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "4a209a58c83b407aad79876883e18244"
format_stamp: "Formatted at 2026-08-06 09:21:55 on dist-test-slave-h3ft"
server_key: "1a0227570c9f399edb2e5f75082d10fc"
server_key_iv: "b6104d1b5abbdab6c108007cbcadf5d1"
server_key_version: "encryptionkey@0"
I20260806 09:21:55.431084 24463 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:21:55.476637 24463 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:21:55.477581 24463 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:21:55.477870 24463 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:21:55.500315 24463 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:43045
I20260806 09:21:55.500336 24560 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:43045 every 8 connection(s)
I20260806 09:21:55.501854 24463 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:21:55.502401 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 24463
I20260806 09:21:55.502648 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:21:55.503633 16730 external_mini_cluster.cc:1428] Setting key 30280d7d26b513b4f104755f22073ad6
I20260806 09:21:55.508308 24563 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:21:55.521744 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:55.505925 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59308 (local address 127.16.86.190:43045)
0806 09:21:55.507276 (+ 1351us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:55.507303 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:55.507376 (+ 73us) server_negotiation.cc:407] Connection header received
0806 09:21:55.509945 (+ 2569us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:55.509976 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:55.510651 (+ 675us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:55.511197 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:21:55.511852 (+ 655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:55.514058 (+ 2206us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:55.515516 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:55.516660 (+ 1144us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:55.516852 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:55.518289 (+ 1437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:21:55.518299 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:21:55.518328 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:21:55.519204 (+ 876us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:21:55.519310 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:21:55.519352 (+ 42us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:21:55.519488 (+ 136us) server_negotiation.cc:1035] Waiting for connection context
0806 09:21:55.519752 (+ 264us) server_negotiation.cc:299] Negotiation successful
0806 09:21:55.520160 (+ 408us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":409,"threads_started":1}
I20260806 09:21:55.524211 24563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244: Bootstrap starting.
I20260806 09:21:55.526783 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:21:55.528470 24563 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244: Neither blocks nor log segments found. Creating new log.
I20260806 09:21:55.529865 24563 log.cc:824] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244: Log is configured to *not* fsync() on all Append() calls
I20260806 09:21:55.533799 24563 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244: No bootstrap required, opened a new log
I20260806 09:21:55.542315 24563 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a209a58c83b407aad79876883e18244" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43045 } }
I20260806 09:21:55.542719 24563 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:21:55.542802 24563 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4a209a58c83b407aad79876883e18244, State: Initialized, Role: FOLLOWER
I20260806 09:21:55.543426 24563 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a209a58c83b407aad79876883e18244" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43045 } }
I20260806 09:21:55.543658 24563 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:21:55.543824 24563 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:21:55.543970 24563 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:21:55.546746 24563 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a209a58c83b407aad79876883e18244" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43045 } }
I20260806 09:21:55.547375 24563 leader_election.cc:302] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4a209a58c83b407aad79876883e18244; no voters:
I20260806 09:21:55.548045 24563 leader_election.cc:288] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:21:55.548318 24568 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:21:55.549355 24568 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [term 1 LEADER]: Becoming Leader. State: Replica: 4a209a58c83b407aad79876883e18244, State: Running, Role: LEADER
I20260806 09:21:55.550041 24568 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a209a58c83b407aad79876883e18244" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43045 } }
I20260806 09:21:55.550683 24563 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:21:55.553371 24569 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4a209a58c83b407aad79876883e18244" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a209a58c83b407aad79876883e18244" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43045 } } }
I20260806 09:21:55.553766 24569 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:55.553071 24570 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4a209a58c83b407aad79876883e18244. Latest consensus state: current_term: 1 leader_uuid: "4a209a58c83b407aad79876883e18244" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4a209a58c83b407aad79876883e18244" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43045 } } }
I20260806 09:21:55.555387 24570 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244 [sys.catalog]: This master's current role is: LEADER
I20260806 09:21:55.559372 24573 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:21:55.566435 24573 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:21:55.577752 24573 catalog_manager.cc:1347] Generated new cluster ID: b4347b0b63a74394b27dffa73ee39641
I20260806 09:21:55.577922 24573 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:21:55.605333 24573 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:21:55.606739 24573 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:21:55.613109 24573 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4a209a58c83b407aad79876883e18244: Generated new TSK 0
I20260806 09:21:55.614634 24573 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:21:55.688050 24566 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:21:55.630525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59312 (local address 127.16.86.190:43045)
0806 09:21:55.630835 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 09:21:55.630843 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:21:55.631324 (+ 481us) server_negotiation.cc:407] Connection header received
0806 09:21:55.631596 (+ 272us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:21:55.631600 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:21:55.631668 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:21:55.631834 (+ 166us) server_negotiation.cc:226] Negotiated authn=JWT
0806 09:21:55.632335 (+ 501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:55.633099 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:55.634466 (+ 1367us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:21:55.635189 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:21:55.635336 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:55.635874 (+ 538us) server_negotiation.cc:365] Received JWT_EXCHANGE NegotiatePB request
0806 09:21:55.687246 (+ 51372us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 09:21:55.687686 (+ 440us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:59312: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":113}
W20260806 09:21:55.688270 24566 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:59312: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
W20260806 09:21:55.688400 24599 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:21:55.630436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43045 (local address 127.0.0.1:59312)
0806 09:21:55.631212 (+ 776us) negotiation.cc:107] Waiting for socket to connect
0806 09:21:55.631231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:21:55.631410 (+ 179us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:21:55.631849 (+ 439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:21:55.631854 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:21:55.631866 (+ 12us) client_negotiation.cc:189] Negotiated authn=JWT
0806 09:21:55.632166 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:55.632175 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:55.633242 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:55.633249 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:55.634322 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:21:55.634332 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:21:55.635393 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:21:55.635402 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:21:55.635550 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:21:55.635585 (+ 35us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
0806 09:21:55.687585 (+ 52000us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
0806 09:21:55.687903 (+ 318us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.16.86.190:43045: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Couldn't resolve host name: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":408,"threads_started":1}
I20260806 09:21:55.689903 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 24463
I20260806 09:21:55.692226 24553 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:21:55.807433 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 24463
2026-08-06T09:21:55Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (887 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (510 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1833 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:21:58 dist-test-slave-h3ft krb5kdc[24622](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:21:58 dist-test-slave-h3ft krb5kdc[24622](info): set up 2 sockets
Aug 06 09:21:58 dist-test-slave-h3ft krb5kdc[24622](info): commencing operation
krb5kdc: starting...
W20260806 09:21:59.477758 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.275s user 0.000s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:21:59 dist-test-slave-h3ft krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:21:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:21:59Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:21:59.791517 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:36175
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:37829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:36175
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:00.171842 24643 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:00.172164 24643 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:00.172334 24643 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:00.179807 24643 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:22:00.179944 24643 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:00.179997 24643 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:00.180099 24643 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:22:00.180202 24643 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:22:00.190577 24643 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:37829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:36175
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:36175
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:00.192090 24643 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:00.193480 24643 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:00.207108 24654 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:00.207597 24650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:00.207306 24649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:00.208585 24643 server_base.cc:1034] running on GCE node
I20260806 09:22:00.209519 24643 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:00.211125 24643 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:00.212383 24643 hybrid_clock.cc:648] HybridClock initialized: now 1786008120212291 us; error 82 us; skew 500 ppm
I20260806 09:22:00.212774 24643 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:00 dist-test-slave-h3ft krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008120, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:00.216459 24643 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:22:00.218344 24643 webserver.cc:460] Webserver started at http://127.16.86.190:41909/ using document root <none> and password file <none>
I20260806 09:22:00.219223 24643 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:00.219381 24643 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:00.219756 24643 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:00.222899 24643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "e397974f64de46c1a2f28770c0d4f5e5"
format_stamp: "Formatted at 2026-08-06 09:22:00 on dist-test-slave-h3ft"
server_key: "cda1e241d098ab80f816b54097eeb5b2"
server_key_iv: "a8556160383d4dc76b0244d6554e34dc"
server_key_version: "encryptionkey@0"
I20260806 09:22:00.223814 24643 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "e397974f64de46c1a2f28770c0d4f5e5"
format_stamp: "Formatted at 2026-08-06 09:22:00 on dist-test-slave-h3ft"
server_key: "cda1e241d098ab80f816b54097eeb5b2"
server_key_iv: "a8556160383d4dc76b0244d6554e34dc"
server_key_version: "encryptionkey@0"
I20260806 09:22:00.229851 24643 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:22:00.234000 24667 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:00.235749 24643 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:22:00.235965 24643 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "e397974f64de46c1a2f28770c0d4f5e5"
format_stamp: "Formatted at 2026-08-06 09:22:00 on dist-test-slave-h3ft"
server_key: "cda1e241d098ab80f816b54097eeb5b2"
server_key_iv: "a8556160383d4dc76b0244d6554e34dc"
server_key_version: "encryptionkey@0"
I20260806 09:22:00.236187 24643 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:00.254150 24643 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:00.259160 24643 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:00.259524 24643 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:00.281724 24643 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:36175
I20260806 09:22:00.281775 24743 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:36175 every 8 connection(s)
I20260806 09:22:00.283325 24643 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:22:00.283946 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 24643
I20260806 09:22:00.284240 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:22:00.285449 16730 external_mini_cluster.cc:1428] Setting key e78bc86bfab281aad23c9f6abdc49f98
I20260806 09:22:00.288488 24746 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:00.302145 24746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5: Bootstrap starting.
Aug 06 09:22:00 dist-test-slave-h3ft krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008119, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:22:00.305797 24746 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:00.307297 24746 log.cc:824] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:00.311218 24746 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5: No bootstrap required, opened a new log
I20260806 09:22:00.314275 24750 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:00.288172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47332 (local address 127.16.86.190:36175)
0806 09:22:00.289213 (+ 1041us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:00.289231 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:00.289288 (+ 57us) server_negotiation.cc:407] Connection header received
0806 09:22:00.291662 (+ 2374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:00.291689 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:00.292235 (+ 546us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:00.292721 (+ 486us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:00.294464 (+ 1743us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:00.296560 (+ 2096us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:00.297988 (+ 1428us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:00.299108 (+ 1120us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:00.299339 (+ 231us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:00.304535 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:00.304546 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:00.304565 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:00.304619 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:00.308999 (+ 4380us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:00.309954 (+ 955us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:00.309964 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:00.309974 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:00.310208 (+ 234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:00.310740 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:00.310745 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:00.310748 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:00.311507 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:00.311841 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:00.312302 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 09:22:00.312845 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":742,"thread_start_us":446,"threads_started":1}
I20260806 09:22:00.320807 24746 raft_consensus.cc:348] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397974f64de46c1a2f28770c0d4f5e5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36175 } }
I20260806 09:22:00.321193 24746 raft_consensus.cc:374] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:00.321259 24746 raft_consensus.cc:729] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e397974f64de46c1a2f28770c0d4f5e5, State: Initialized, Role: FOLLOWER
I20260806 09:22:00.321831 24746 consensus_queue.cc:260] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397974f64de46c1a2f28770c0d4f5e5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36175 } }
I20260806 09:22:00.322039 24746 raft_consensus.cc:388] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:22:00.322214 24746 raft_consensus.cc:482] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:22:00.322405 24746 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:00.324887 24746 raft_consensus.cc:504] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397974f64de46c1a2f28770c0d4f5e5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36175 } }
I20260806 09:22:00.325410 24746 leader_election.cc:302] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e397974f64de46c1a2f28770c0d4f5e5; no voters:
I20260806 09:22:00.326164 24746 leader_election.cc:288] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:22:00.326472 24753 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:00.327615 24753 raft_consensus.cc:686] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [term 1 LEADER]: Becoming Leader. State: Replica: e397974f64de46c1a2f28770c0d4f5e5, State: Running, Role: LEADER
I20260806 09:22:00.328679 24746 sys_catalog.cc:564] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:22:00.328534 24753 consensus_queue.cc:237] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397974f64de46c1a2f28770c0d4f5e5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36175 } }
I20260806 09:22:00.332777 24754 sys_catalog.cc:455] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e397974f64de46c1a2f28770c0d4f5e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397974f64de46c1a2f28770c0d4f5e5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36175 } } }
I20260806 09:22:00.333410 24754 sys_catalog.cc:458] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:00.333828 24755 sys_catalog.cc:455] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e397974f64de46c1a2f28770c0d4f5e5. Latest consensus state: current_term: 1 leader_uuid: "e397974f64de46c1a2f28770c0d4f5e5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e397974f64de46c1a2f28770c0d4f5e5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36175 } } }
I20260806 09:22:00.334148 24755 sys_catalog.cc:458] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:00.338621 24765 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:22:00.345079 24765 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:22:00.356057 24765 catalog_manager.cc:1347] Generated new cluster ID: 130c04e9b846419480486d46bcfbbecb
I20260806 09:22:00.356230 24765 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:22:00.371181 24765 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:22:00.373052 24765 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:22:00.386852 24765 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e397974f64de46c1a2f28770c0d4f5e5: Generated new TSK 0
I20260806 09:22:00.388078 24765 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:22:00.480557 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36175
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:37829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:00.870807 24778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:00.871213 24778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:00.871321 24778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:00.878728 24778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:00.878861 24778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:00.878983 24778 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:22:00.889357 24778 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:37829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:00.890792 24778 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:00.892195 24778 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:00.907855 24795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:00.907785 24798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:00.907893 24793 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:00.909826 24778 server_base.cc:1034] running on GCE node
I20260806 09:22:00.910549 24778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:00.911715 24778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:00.912986 24778 hybrid_clock.cc:648] HybridClock initialized: now 1786008120912927 us; error 54 us; skew 500 ppm
I20260806 09:22:00.913354 24778 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:00 dist-test-slave-h3ft krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008120, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:00.917021 24778 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:22:00.918809 24778 webserver.cc:460] Webserver started at http://127.16.86.129:41531/ using document root <none> and password file <none>
I20260806 09:22:00.919622 24778 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:00.919783 24778 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:00.920135 24778 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:00.923283 24778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "6aaa2f67d5af4e659e8f60c89933301d"
format_stamp: "Formatted at 2026-08-06 09:22:00 on dist-test-slave-h3ft"
server_key: "5286a95a6b30a342f457b867f3a89353"
server_key_iv: "3b69d667fce3150391167e5c1bbab774"
server_key_version: "encryptionkey@0"
I20260806 09:22:00.924121 24778 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "6aaa2f67d5af4e659e8f60c89933301d"
format_stamp: "Formatted at 2026-08-06 09:22:00 on dist-test-slave-h3ft"
server_key: "5286a95a6b30a342f457b867f3a89353"
server_key_iv: "3b69d667fce3150391167e5c1bbab774"
server_key_version: "encryptionkey@0"
I20260806 09:22:00.930467 24778 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:22:00.935067 24811 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:00.937179 24778 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 09:22:00.937462 24778 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "6aaa2f67d5af4e659e8f60c89933301d"
format_stamp: "Formatted at 2026-08-06 09:22:00 on dist-test-slave-h3ft"
server_key: "5286a95a6b30a342f457b867f3a89353"
server_key_iv: "3b69d667fce3150391167e5c1bbab774"
server_key_version: "encryptionkey@0"
I20260806 09:22:00.937808 24778 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:00.954027 24778 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:00.958909 24778 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:00.959313 24778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:00.960439 24778 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:00.962077 24778 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:00.962177 24778 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:00.962281 24778 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:00.962337 24778 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:01.004004 24778 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:43995
I20260806 09:22:01.004046 24975 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:43995 every 8 connection(s)
I20260806 09:22:01.005743 24778 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:22:01.007678 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 24778
I20260806 09:22:01.007913 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:22:01.008865 16730 external_mini_cluster.cc:1428] Setting key 78ac8370411a8968de7d924dd982b979
Aug 06 09:22:01 dist-test-slave-h3ft krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008120, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:22:01.033196 24978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:01.009547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:36483 (local address 127.16.86.190:36175)
0806 09:22:01.010529 (+ 982us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:01.010536 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:01.011775 (+ 1239us) server_negotiation.cc:407] Connection header received
0806 09:22:01.013988 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:01.013995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:01.014105 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:01.014234 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:01.017056 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:01.017891 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:01.020874 (+ 2983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:01.021340 (+ 466us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:01.021451 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:01.026195 (+ 4744us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:01.026200 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:01.026204 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:01.026243 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:01.029358 (+ 3115us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:01.030266 (+ 908us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:01.030272 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:01.030276 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:01.030463 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:01.030949 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:01.030954 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:01.030958 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:01.031357 (+ 399us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:01.031531 (+ 174us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:01.032426 (+ 895us) server_negotiation.cc:299] Negotiation successful
0806 09:22:01.032718 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":795,"thread_start_us":616,"threads_started":1}
I20260806 09:22:01.034613 24981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:01.010278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:36175 (local address 127.16.86.129:36483)
0806 09:22:01.011533 (+ 1255us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:01.011604 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:01.013643 (+ 2039us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:01.014685 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:01.014704 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:01.015794 (+ 1090us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:01.016797 (+ 1003us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:01.016831 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:01.018105 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:01.018114 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:01.020670 (+ 2556us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:01.020692 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:01.021618 (+ 926us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:01.021629 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:01.022013 (+ 384us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:01.023091 (+ 1078us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:01.023134 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:01.025919 (+ 2785us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:01.029532 (+ 3613us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:01.029544 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:01.029574 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:01.030119 (+ 545us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:01.030583 (+ 464us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:01.030588 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:01.030592 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:01.030813 (+ 221us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:01.031504 (+ 691us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:01.031514 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:01.031996 (+ 482us) client_negotiation.cc:769] Sending connection context
0806 09:22:01.032348 (+ 352us) client_negotiation.cc:240] Negotiation successful
0806 09:22:01.033051 (+ 703us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":814,"thread_start_us":406,"threads_started":1}
I20260806 09:22:01.036780 24976 heartbeater.cc:347] Connected to a master server at 127.16.86.190:36175
I20260806 09:22:01.037186 24976 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:01.038115 24976 heartbeater.cc:511] Master 127.16.86.190:36175 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
I20260806 09:22:01.041080 24689 ts_manager.cc:194] Registered new tserver with Master: 6aaa2f67d5af4e659e8f60c89933301d (127.16.86.129:43995)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
I20260806 09:22:01.043623 24689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:36483
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:22:01.079386 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36175
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:37829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:01.436285 24986 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:01.436658 24986 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:01.436846 24986 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:01.444842 24986 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:01.445007 24986 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:01.445207 24986 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:22:01.455508 24986 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:37829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:01.457006 24986 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:01.458455 24986 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:01.473899 24996 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:01.474030 25000 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:01.474059 24997 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:01.475247 24986 server_base.cc:1034] running on GCE node
I20260806 09:22:01.475977 24986 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:01.477061 24986 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:01.478353 24986 hybrid_clock.cc:648] HybridClock initialized: now 1786008121478273 us; error 66 us; skew 500 ppm
I20260806 09:22:01.478762 24986 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:01 dist-test-slave-h3ft krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008121, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:01.483311 24986 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:22:01.485123 24986 webserver.cc:460] Webserver started at http://127.16.86.130:45213/ using document root <none> and password file <none>
I20260806 09:22:01.485915 24986 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:01.486083 24986 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:01.486438 24986 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:01.489814 24986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "a5d9936dc2a54002b8992f3731428bf5"
format_stamp: "Formatted at 2026-08-06 09:22:01 on dist-test-slave-h3ft"
server_key: "fb567fe16ccd31581b5841d8340d5b4e"
server_key_iv: "e065d990cd2accfc9dec21be88d67dad"
server_key_version: "encryptionkey@0"
I20260806 09:22:01.490711 24986 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "a5d9936dc2a54002b8992f3731428bf5"
format_stamp: "Formatted at 2026-08-06 09:22:01 on dist-test-slave-h3ft"
server_key: "fb567fe16ccd31581b5841d8340d5b4e"
server_key_iv: "e065d990cd2accfc9dec21be88d67dad"
server_key_version: "encryptionkey@0"
I20260806 09:22:01.497848 24986 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 09:22:01.502707 25007 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:01.504812 24986 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:22:01.505121 24986 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "a5d9936dc2a54002b8992f3731428bf5"
format_stamp: "Formatted at 2026-08-06 09:22:01 on dist-test-slave-h3ft"
server_key: "fb567fe16ccd31581b5841d8340d5b4e"
server_key_iv: "e065d990cd2accfc9dec21be88d67dad"
server_key_version: "encryptionkey@0"
I20260806 09:22:01.505383 24986 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:01.548817 24986 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:01.553959 24986 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:01.554320 24986 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:01.555500 24986 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:01.557164 24986 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:01.557283 24986 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:01.557461 24986 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:01.557540 24986 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:01.601164 24986 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:36759
I20260806 09:22:01.601270 25159 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:36759 every 8 connection(s)
I20260806 09:22:01.602635 24986 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:22:01.605214 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 24986
I20260806 09:22:01.605484 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:22:01.606544 16730 external_mini_cluster.cc:1428] Setting key d17c55cb46e71b7231726bf21e277164
Aug 06 09:22:01 dist-test-slave-h3ft krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008121, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:22:01.628804 25162 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:01.606476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:34427 (local address 127.16.86.190:36175)
0806 09:22:01.607229 (+ 753us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:01.607237 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:01.608492 (+ 1255us) server_negotiation.cc:407] Connection header received
0806 09:22:01.611089 (+ 2597us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:01.611095 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:01.611201 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:01.611326 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:01.613823 (+ 2497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:01.614674 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:01.616670 (+ 1996us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:01.617145 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:01.617297 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:01.621929 (+ 4632us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:01.621939 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:01.621943 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:01.622004 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:01.625242 (+ 3238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:01.626128 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:01.626138 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:01.626142 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:01.626285 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:01.626766 (+ 481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:01.626775 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:01.626778 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:01.627154 (+ 376us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:01.627323 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:01.628108 (+ 785us) server_negotiation.cc:299] Negotiation successful
0806 09:22:01.628379 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":364,"threads_started":1}
I20260806 09:22:01.629992 25163 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:01.607131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:36175 (local address 127.16.86.130:34427)
0806 09:22:01.608276 (+ 1145us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:01.608334 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:01.610733 (+ 2399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:01.611620 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:01.611719 (+ 99us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:01.612503 (+ 784us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:01.613479 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:01.613514 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:01.614864 (+ 1350us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:01.614868 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:01.616466 (+ 1598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:01.616482 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:01.617414 (+ 932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:01.617421 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:01.617685 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:01.618465 (+ 780us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:01.618490 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:01.621428 (+ 2938us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:01.625435 (+ 4007us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:01.625446 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:01.625468 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:01.625976 (+ 508us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:01.626405 (+ 429us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:01.626410 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:01.626414 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:01.626633 (+ 219us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:01.627313 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:01.627322 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:01.627699 (+ 377us) client_negotiation.cc:769] Sending connection context
0806 09:22:01.628004 (+ 305us) client_negotiation.cc:240] Negotiation successful
0806 09:22:01.628598 (+ 594us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":537,"threads_started":1}
I20260806 09:22:01.632243 25160 heartbeater.cc:347] Connected to a master server at 127.16.86.190:36175
I20260806 09:22:01.632663 25160 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:01.633699 25160 heartbeater.cc:511] Master 127.16.86.190:36175 requested a full tablet report, sending...
I20260806 09:22:01.635902 24689 ts_manager.cc:194] Registered new tserver with Master: a5d9936dc2a54002b8992f3731428bf5 (127.16.86.130:36759)
I20260806 09:22:01.637423 24689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:34427
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:22:01.684286 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36175
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:37829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:22:02.047811 24976 heartbeater.cc:503] Master 127.16.86.190:36175 was elected leader, sending a full tablet report...
W20260806 09:22:02.120321 25166 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:02.120795 25166 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:02.120963 25166 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:02.129417 25166 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:02.129592 25166 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:02.129805 25166 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:22:02.141829 25166 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:37829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36175
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:02.144014 25166 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:02.145857 25166 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:02.163884 25172 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:02.164019 25171 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:02.164000 25174 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:02.164832 25166 server_base.cc:1034] running on GCE node
I20260806 09:22:02.165690 25166 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:02.166980 25166 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:02.168318 25166 hybrid_clock.cc:648] HybridClock initialized: now 1786008122168267 us; error 55 us; skew 500 ppm
I20260806 09:22:02.168726 25166 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:02.172780 25166 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:22:02.174875 25166 webserver.cc:460] Webserver started at http://127.16.86.131:39311/ using document root <none> and password file <none>
I20260806 09:22:02.175841 25166 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:02.176015 25166 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:02.176446 25166 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:02.180315 25166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "f3ca3e436de04da9ab9b319d83f1b3d3"
format_stamp: "Formatted at 2026-08-06 09:22:02 on dist-test-slave-h3ft"
server_key: "a5d7745639b88c0f70020dd1743ccb49"
server_key_iv: "3433f0e938994a4003ee935aee7921ab"
server_key_version: "encryptionkey@0"
I20260806 09:22:02.181275 25166 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "f3ca3e436de04da9ab9b319d83f1b3d3"
format_stamp: "Formatted at 2026-08-06 09:22:02 on dist-test-slave-h3ft"
server_key: "a5d7745639b88c0f70020dd1743ccb49"
server_key_iv: "3433f0e938994a4003ee935aee7921ab"
server_key_version: "encryptionkey@0"
I20260806 09:22:02.188724 25166 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260806 09:22:02.193645 25183 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:02.195592 25166 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:22:02.195858 25166 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "f3ca3e436de04da9ab9b319d83f1b3d3"
format_stamp: "Formatted at 2026-08-06 09:22:02 on dist-test-slave-h3ft"
server_key: "a5d7745639b88c0f70020dd1743ccb49"
server_key_iv: "3433f0e938994a4003ee935aee7921ab"
server_key_version: "encryptionkey@0"
I20260806 09:22:02.196190 25166 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:02.234668 25166 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:02.240674 25166 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:02.241063 25166 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:02.242462 25166 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:02.244606 25166 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:02.244722 25166 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:02.244904 25166 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:02.244990 25166 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:02.290510 25166 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:38939
I20260806 09:22:02.290690 25356 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:38939 every 8 connection(s)
I20260806 09:22:02.292645 25166 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:22:02.301218 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 25166
I20260806 09:22:02.301453 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:22:02.302006 16730 external_mini_cluster.cc:1428] Setting key 8ffd5e7c1392a6255a2827fb5e16e163
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:22:02.319381 25360 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:02.296359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:37289 (local address 127.16.86.190:36175)
0806 09:22:02.297001 (+ 642us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:02.297008 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:02.298356 (+ 1348us) server_negotiation.cc:407] Connection header received
0806 09:22:02.300699 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:02.300708 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:02.300862 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:02.301039 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:02.303400 (+ 2361us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.304549 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.306332 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.306796 (+ 464us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.306944 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:02.312415 (+ 5471us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:02.312422 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:02.312425 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:02.312470 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:02.315639 (+ 3169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.316543 (+ 904us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.316554 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.316557 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.316687 (+ 130us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.317180 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.317190 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.317192 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.317566 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:02.317752 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:02.318616 (+ 864us) server_negotiation.cc:299] Negotiation successful
0806 09:22:02.318878 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":478,"thread_start_us":303,"threads_started":1}
I20260806 09:22:02.320827 25361 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:02.296996 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:36175 (local address 127.16.86.131:37289)
0806 09:22:02.298118 (+ 1122us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:02.298176 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:02.300242 (+ 2066us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:02.301346 (+ 1104us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:02.301358 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:02.302324 (+ 966us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:02.303172 (+ 848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:02.303197 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.304736 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.304741 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:02.306162 (+ 1421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:02.306177 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.307125 (+ 948us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.307135 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:02.307431 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:02.308268 (+ 837us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:02.308301 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:02.312094 (+ 3793us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:02.315820 (+ 3726us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:02.315831 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:02.315855 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:02.316387 (+ 532us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:02.316810 (+ 423us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:02.316815 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:02.316818 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:02.317046 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:02.317715 (+ 669us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:02.317725 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:02.318179 (+ 454us) client_negotiation.cc:769] Sending connection context
0806 09:22:02.318513 (+ 334us) client_negotiation.cc:240] Negotiation successful
0806 09:22:02.319238 (+ 725us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":759,"thread_start_us":422,"threads_started":1}
I20260806 09:22:02.322877 25357 heartbeater.cc:347] Connected to a master server at 127.16.86.190:36175
I20260806 09:22:02.323340 25357 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:02.324514 25357 heartbeater.cc:511] Master 127.16.86.190:36175 requested a full tablet report, sending...
I20260806 09:22:02.326395 24689 ts_manager.cc:194] Registered new tserver with Master: f3ca3e436de04da9ab9b319d83f1b3d3 (127.16.86.131:38939)
I20260806 09:22:02.327541 24689 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:37289
I20260806 09:22:02.332399 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/test-admin.keytab.
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:02.376001 16730 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20260806 09:22:02.383174 25366 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:02.379861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34510 (local address 127.16.86.129:43995)
0806 09:22:02.380751 (+ 890us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:02.380759 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:02.380792 (+ 33us) server_negotiation.cc:407] Connection header received
0806 09:22:02.380863 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:02.380871 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:02.381105 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:02.381262 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:02.382693 (+ 1431us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34510: BlockingRecv error: recv got EOF from 127.0.0.1:34510 (error 108)
Metrics: {"server-negotiator.queue_time_us":692,"thread_start_us":424,"threads_started":1}
W20260806 09:22:02.383215 16730 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/krb5cc' of type FILE
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:02.385387 16730 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:02.387507 16730 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:22:02.409924 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:02.389933 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34514 (local address 127.16.86.129:43995)
0806 09:22:02.390291 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:02.390299 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:02.390657 (+ 358us) server_negotiation.cc:407] Connection header received
0806 09:22:02.390889 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:02.390894 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:02.390975 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:02.391174 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:02.392508 (+ 1334us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.393827 (+ 1319us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.395712 (+ 1885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.396412 (+ 700us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.396550 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:02.401879 (+ 5329us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:02.401895 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:02.401913 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:02.401957 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:02.405406 (+ 3449us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.406701 (+ 1295us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.406715 (+ 14us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.406718 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.406938 (+ 220us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.407582 (+ 644us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.407587 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.407589 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.408110 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:02.408412 (+ 302us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:02.408897 (+ 485us) server_negotiation.cc:299] Negotiation successful
0806 09:22:02.409581 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
W20260806 09:22:02.417956 25366 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:02.415135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34524 (local address 127.16.86.129:43995)
0806 09:22:02.415499 (+ 364us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:02.415509 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:02.415904 (+ 395us) server_negotiation.cc:407] Connection header received
0806 09:22:02.416172 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:02.416178 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:02.416276 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:02.416413 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:02.417517 (+ 1104us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34524: BlockingRecv error: recv got EOF from 127.0.0.1:34524 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260806 09:22:02.418725 16730 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/krb5cc)
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:02.420950 16730 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:02.422724 16730 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:22:02.441980 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:02.424725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34532 (local address 127.16.86.129:43995)
0806 09:22:02.425094 (+ 369us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:02.425103 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:02.425320 (+ 217us) server_negotiation.cc:407] Connection header received
0806 09:22:02.425524 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:02.425528 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:02.425608 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:02.425717 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:02.426963 (+ 1246us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.427826 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.429261 (+ 1435us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.429845 (+ 584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.429966 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:02.434791 (+ 4825us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:02.434796 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:02.434800 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:02.434852 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:02.438558 (+ 3706us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.439556 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.439560 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.439563 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.439729 (+ 166us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.440259 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.440263 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.440265 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.440565 (+ 300us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:02.440816 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:02.441295 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 09:22:02.441564 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
W20260806 09:22:02.447893 25366 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:02.445618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34534 (local address 127.16.86.129:43995)
0806 09:22:02.445807 (+ 189us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:02.445812 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:02.446240 (+ 428us) server_negotiation.cc:407] Connection header received
0806 09:22:02.446383 (+ 143us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:02.446387 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:02.446494 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:02.446593 (+ 99us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:02.447608 (+ 1015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34534: BlockingRecv error: recv got EOF from 127.0.0.1:34534 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
W20260806 09:22:02.448124 16730 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1786008061536593-16730-0/krb5kdc/krb5cc' of type FILE
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:02.450042 16730 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:02.451591 16730 init.cc:295] Successfully reacquired a new kerberos TGT
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:22:02.471439 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:02.453572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34542 (local address 127.16.86.129:43995)
0806 09:22:02.454014 (+ 442us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:02.454021 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:02.454443 (+ 422us) server_negotiation.cc:407] Connection header received
0806 09:22:02.454680 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:02.454685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:02.454771 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:02.454892 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:02.456538 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.457377 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.459275 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.459802 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.459941 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:02.464904 (+ 4963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:02.464910 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:02.464913 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:02.464969 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:02.468191 (+ 3222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.469119 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.469124 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.469127 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.469285 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.469770 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.469774 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.469776 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.470089 (+ 313us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:02.470351 (+ 262us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:02.470730 (+ 379us) server_negotiation.cc:299] Negotiation successful
0806 09:22:02.471061 (+ 331us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20260806 09:22:02.473730 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 24778
I20260806 09:22:02.489703 25366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:02.474834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34550 (local address 127.16.86.129:43995)
0806 09:22:02.475178 (+ 344us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:02.475184 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:02.475488 (+ 304us) server_negotiation.cc:407] Connection header received
0806 09:22:02.475627 (+ 139us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:02.475632 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:02.475714 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:02.475835 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:02.477072 (+ 1237us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.477911 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.479417 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.479951 (+ 534us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.480103 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:02.482773 (+ 2670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:02.482779 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:02.482783 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:02.482835 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:02.486113 (+ 3278us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.487207 (+ 1094us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.487213 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.487216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.487411 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.487910 (+ 499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.487914 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.487916 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.488290 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:02.488582 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:02.489051 (+ 469us) server_negotiation.cc:299] Negotiation successful
0806 09:22:02.489305 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 09:22:02.490691 24967 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:02.586417 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 24778
I20260806 09:22:02.616622 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 24986
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:22:02.637632 25383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:02.618054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58706 (local address 127.16.86.130:36759)
0806 09:22:02.618863 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:02.618872 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:02.618906 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:22:02.618991 (+ 85us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:02.619055 (+ 64us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:02.619373 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:02.619550 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:02.620771 (+ 1221us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.622097 (+ 1326us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.623890 (+ 1793us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.624674 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.624814 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:02.629281 (+ 4467us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:02.629302 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:02.629325 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:02.629392 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:02.633626 (+ 4234us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.634838 (+ 1212us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.634850 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.634854 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.635143 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.635647 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.635652 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.635654 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.636144 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:02.636475 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:02.636823 (+ 348us) server_negotiation.cc:299] Negotiation successful
0806 09:22:02.637203 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":616,"thread_start_us":417,"threads_started":1}
I20260806 09:22:02.639397 25154 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:02.728410 25160 heartbeater.cc:503] Master 127.16.86.190:36175 was elected leader, sending a full tablet report...
I20260806 09:22:02.729940 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 24986
W20260806 09:22:02.761994 24669 connection.cc:535] server connection from 127.16.86.130:34427 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 09:22:02.762539 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 25166
Aug 06 09:22:02 dist-test-slave-h3ft krb5kdc[24622](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008122, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:22:02.782709 25385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:02.763689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55122 (local address 127.16.86.131:38939)
0806 09:22:02.764278 (+ 589us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:02.764285 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:02.764321 (+ 36us) server_negotiation.cc:407] Connection header received
0806 09:22:02.764393 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:02.764400 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:02.764630 (+ 230us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:02.764794 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:02.766163 (+ 1369us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.767330 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.768963 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:02.769608 (+ 645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:02.769751 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:02.775268 (+ 5517us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:02.775285 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:02.775302 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:02.775345 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:02.778717 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.779829 (+ 1112us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.779837 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.779840 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.780011 (+ 171us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:02.780486 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:02.780491 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:02.780493 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:02.781275 (+ 782us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:02.781809 (+ 534us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:02.781977 (+ 168us) server_negotiation.cc:299] Negotiation successful
0806 09:22:02.782309 (+ 332us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":414,"thread_start_us":274,"threads_started":1}
I20260806 09:22:02.784487 25346 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:02.886994 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 25166
I20260806 09:22:02.916996 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 24643
I20260806 09:22:02.919241 24738 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:03.050618 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 24643
2026-08-06T09:22:03Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4909 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:22:03 dist-test-slave-h3ft krb5kdc[25392](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:22:03 dist-test-slave-h3ft krb5kdc[25392](info): set up 2 sockets
Aug 06 09:22:03 dist-test-slave-h3ft krb5kdc[25392](info): commencing operation
krb5kdc: starting...
W20260806 09:22:04.473376 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.360s user 0.000s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:22:04 dist-test-slave-h3ft krb5kdc[25392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:22:04Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:22:04Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.190.keytab.
Entry for principal oryx/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.190.keytab.
I20260806 09:22:04.837169 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:43733
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.190.keytab
--principal=oryx/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:43733
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:05.302125 25413 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:05.302589 25413 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:05.302702 25413 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:05.311311 25413 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:22:05.311554 25413 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:05.311679 25413 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:05.311798 25413 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:22:05.311885 25413 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:22:05.323925 25413 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:43733
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.190.keytab
--principal=oryx/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:43733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:05.325913 25413 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:05.327905 25413 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:05.345892 25426 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:05.345798 25431 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:05.346554 25423 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:05.348111 25413 server_base.cc:1034] running on GCE node
I20260806 09:22:05.349464 25413 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:05.351724 25413 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:05.353106 25413 hybrid_clock.cc:648] HybridClock initialized: now 1786008125352998 us; error 84 us; skew 500 ppm
I20260806 09:22:05.353682 25413 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:05 dist-test-slave-h3ft krb5kdc[25392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008125, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:05.358666 25413 init.cc:377] Logged in from keytab as oryx/127.16.86.190@KRBTEST.COM (short username oryx)
I20260806 09:22:05.361106 25413 webserver.cc:460] Webserver started at http://127.16.86.190:39381/ using document root <none> and password file <none>
I20260806 09:22:05.362109 25413 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:05.362303 25413 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:05.362713 25413 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:05.367115 25413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "4b2fc5f140794fee87c356b29cd9fb6a"
format_stamp: "Formatted at 2026-08-06 09:22:05 on dist-test-slave-h3ft"
server_key: "3637ac12288fc1c6bac9118948275aba"
server_key_iv: "07f85ae7a12246812f3ae4fafa576364"
server_key_version: "encryptionkey@0"
I20260806 09:22:05.368260 25413 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "4b2fc5f140794fee87c356b29cd9fb6a"
format_stamp: "Formatted at 2026-08-06 09:22:05 on dist-test-slave-h3ft"
server_key: "3637ac12288fc1c6bac9118948275aba"
server_key_iv: "07f85ae7a12246812f3ae4fafa576364"
server_key_version: "encryptionkey@0"
I20260806 09:22:05.375686 25413 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260806 09:22:05.381151 25440 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:05.383489 25413 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:22:05.383786 25413 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "4b2fc5f140794fee87c356b29cd9fb6a"
format_stamp: "Formatted at 2026-08-06 09:22:05 on dist-test-slave-h3ft"
server_key: "3637ac12288fc1c6bac9118948275aba"
server_key_iv: "07f85ae7a12246812f3ae4fafa576364"
server_key_version: "encryptionkey@0"
I20260806 09:22:05.384099 25413 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:05.429886 25413 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:05.435777 25413 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:05.436179 25413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:05.461735 25413 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:43733
I20260806 09:22:05.461784 25514 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:43733 every 8 connection(s)
I20260806 09:22:05.463698 25413 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:22:05.468832 25517 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:05.471455 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 25413
I20260806 09:22:05.471694 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:22:05.472565 16730 external_mini_cluster.cc:1428] Setting key 1c1d863802a5ebec90e33ba3620d7090
I20260806 09:22:05.486039 25517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a: Bootstrap starting.
I20260806 09:22:05.491276 25517 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:05.493273 25517 log.cc:824] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:22:05 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:05.498198 25517 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a: No bootstrap required, opened a new log
I20260806 09:22:05.508322 25520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:05.475714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43642 (local address 127.16.86.190:43733)
0806 09:22:05.476955 (+ 1241us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:05.476973 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:05.477037 (+ 64us) server_negotiation.cc:407] Connection header received
0806 09:22:05.479576 (+ 2539us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:05.479616 (+ 40us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:05.480277 (+ 661us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:05.480772 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:05.484272 (+ 3500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:05.486133 (+ 1861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:05.488507 (+ 2374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:05.489758 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:05.489981 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:05.496671 (+ 6690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:05.496686 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:05.496713 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:05.496778 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:05.502020 (+ 5242us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:05.503107 (+ 1087us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:05.503124 (+ 17us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:05.503136 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:05.503441 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:05.503938 (+ 497us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:05.503946 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:05.503951 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:05.504915 (+ 964us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:05.505327 (+ 412us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:05.505792 (+ 465us) server_negotiation.cc:299] Negotiation successful
0806 09:22:05.506439 (+ 647us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":920,"thread_start_us":573,"threads_started":1}
I20260806 09:22:05.511096 25517 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } }
I20260806 09:22:05.511654 25517 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:05.511801 25517 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2fc5f140794fee87c356b29cd9fb6a, State: Initialized, Role: FOLLOWER
I20260806 09:22:05.512743 25517 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } }
I20260806 09:22:05.513108 25517 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:22:05.513284 25517 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:22:05.513495 25517 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:05.516235 25517 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } }
I20260806 09:22:05.516882 25517 leader_election.cc:302] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b2fc5f140794fee87c356b29cd9fb6a; no voters:
I20260806 09:22:05.517665 25517 leader_election.cc:288] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:22:05.517964 25522 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:05.519999 25522 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 1 LEADER]: Becoming Leader. State: Replica: 4b2fc5f140794fee87c356b29cd9fb6a, State: Running, Role: LEADER
I20260806 09:22:05.520694 25522 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } }
I20260806 09:22:05.521107 25517 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:22:05.524375 25525 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b2fc5f140794fee87c356b29cd9fb6a. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } }
I20260806 09:22:05.525547 25525 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:05.524665 25523 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } }
I20260806 09:22:05.525714 25523 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:05.529641 25540 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:22:05.536073 25540 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:22:05.548061 25540 catalog_manager.cc:1347] Generated new cluster ID: 4c0afa372dac4077b1e53622c7454eba
I20260806 09:22:05.548250 25540 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:22:05.558209 25540 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:22:05.560124 25540 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:22:05.575989 25540 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a: Generated new TSK 0
I20260806 09:22:05.577767 25540 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab.
Entry for principal oryx/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab.
I20260806 09:22:05.666730 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43733
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab
--principal=oryx/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:06.118108 25558 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:06.118570 25558 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:06.118757 25558 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:06.127602 25558 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:06.127846 25558 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:06.128065 25558 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:22:06.140158 25558 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab
--principal=oryx/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:06.142220 25558 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:06.144061 25558 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:06.163664 25567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:06.163708 25566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:06.163762 25569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:06.164830 25558 server_base.cc:1034] running on GCE node
I20260806 09:22:06.165784 25558 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:06.167296 25558 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:06.168569 25558 hybrid_clock.cc:648] HybridClock initialized: now 1786008126168463 us; error 95 us; skew 500 ppm
I20260806 09:22:06.169034 25558 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:06 dist-test-slave-h3ft krb5kdc[25392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:06.173477 25558 init.cc:377] Logged in from keytab as oryx/127.16.86.129@KRBTEST.COM (short username oryx)
I20260806 09:22:06.176100 25558 webserver.cc:460] Webserver started at http://127.16.86.129:41761/ using document root <none> and password file <none>
I20260806 09:22:06.177135 25558 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:06.177276 25558 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:06.177693 25558 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:06.181851 25558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "d3d9877aba6a445280ab89df34417458"
format_stamp: "Formatted at 2026-08-06 09:22:06 on dist-test-slave-h3ft"
server_key: "715b4dc5ad435655b47d19509ea38ccf"
server_key_iv: "5baa3e35ef30225a988760d9673a5eda"
server_key_version: "encryptionkey@0"
I20260806 09:22:06.183130 25558 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "d3d9877aba6a445280ab89df34417458"
format_stamp: "Formatted at 2026-08-06 09:22:06 on dist-test-slave-h3ft"
server_key: "715b4dc5ad435655b47d19509ea38ccf"
server_key_iv: "5baa3e35ef30225a988760d9673a5eda"
server_key_version: "encryptionkey@0"
I20260806 09:22:06.191147 25558 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260806 09:22:06.196470 25578 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:06.198594 25558 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.002s
I20260806 09:22:06.198890 25558 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "d3d9877aba6a445280ab89df34417458"
format_stamp: "Formatted at 2026-08-06 09:22:06 on dist-test-slave-h3ft"
server_key: "715b4dc5ad435655b47d19509ea38ccf"
server_key_iv: "5baa3e35ef30225a988760d9673a5eda"
server_key_version: "encryptionkey@0"
I20260806 09:22:06.199293 25558 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:06.218631 25558 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:06.223855 25558 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:06.224196 25558 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:06.225389 25558 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:06.227162 25558 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:06.227250 25558 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:06.227385 25558 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:06.227466 25558 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:06.271924 25558 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:45867
I20260806 09:22:06.272133 25737 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:45867 every 8 connection(s)
I20260806 09:22:06.273538 25558 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:22:06.275594 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 25558
I20260806 09:22:06.275870 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:22:06.276743 16730 external_mini_cluster.cc:1428] Setting key 5b7167ef87697c7f9e57337ab489a6e5
Aug 06 09:22:06 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:06.302246 25740 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:06.278241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:42589 (local address 127.16.86.190:43733)
0806 09:22:06.278881 (+ 640us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:06.278894 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:06.280728 (+ 1834us) server_negotiation.cc:407] Connection header received
0806 09:22:06.283253 (+ 2525us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:06.283261 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:06.283369 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:06.283506 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:06.285642 (+ 2136us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:06.286816 (+ 1174us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:06.288894 (+ 2078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:06.289625 (+ 731us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:06.289798 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:06.295385 (+ 5587us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:06.295391 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:06.295395 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:06.295435 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:06.298553 (+ 3118us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:06.299635 (+ 1082us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:06.299647 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:06.299652 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:06.299812 (+ 160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:06.300321 (+ 509us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:06.300325 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:06.300327 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:06.300617 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:06.300784 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:06.301583 (+ 799us) server_negotiation.cc:299] Negotiation successful
0806 09:22:06.301842 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":479,"thread_start_us":334,"threads_started":1}
I20260806 09:22:06.303407 25741 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:06.279239 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.129:42589)
0806 09:22:06.280477 (+ 1238us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:06.280542 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:06.282783 (+ 2241us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:06.283805 (+ 1022us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:06.283817 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:06.284587 (+ 770us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:06.285392 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:06.285419 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:06.287125 (+ 1706us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:06.287129 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:06.288666 (+ 1537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:06.288681 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:06.289950 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:06.289962 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:06.290274 (+ 312us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:06.291210 (+ 936us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:06.291239 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:06.295083 (+ 3844us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:06.298781 (+ 3698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:06.298793 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:06.298824 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:06.299477 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:06.299937 (+ 460us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:06.299942 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:06.299946 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:06.300166 (+ 220us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:06.300757 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:06.300766 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:06.301187 (+ 421us) client_negotiation.cc:769] Sending connection context
0806 09:22:06.301472 (+ 285us) client_negotiation.cc:240] Negotiation successful
0806 09:22:06.302095 (+ 623us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":805,"thread_start_us":514,"threads_started":1}
I20260806 09:22:06.305652 25738 heartbeater.cc:347] Connected to a master server at 127.16.86.190:43733
I20260806 09:22:06.306245 25738 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:06.307721 25738 heartbeater.cc:511] Master 127.16.86.190:43733 requested a full tablet report, sending...
WARNING: no policy specified for oryx/127.16.86.130@KRBTEST.COM; defaulting to no policy
I20260806 09:22:06.311733 25460 ts_manager.cc:194] Registered new tserver with Master: d3d9877aba6a445280ab89df34417458 (127.16.86.129:45867)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.130@KRBTEST.COM" created.
I20260806 09:22:06.313547 25460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:42589
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab.
Entry for principal oryx/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab.
I20260806 09:22:06.357128 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43733
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab
--principal=oryx/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:06.798954 25745 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:06.799492 25745 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:06.799624 25745 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:06.808653 25745 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:06.808876 25745 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:06.809034 25745 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:22:06.821327 25745 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab
--principal=oryx/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:06.823411 25745 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:06.825220 25745 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:06.844130 25750 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:06.843962 25751 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:06.844508 25754 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:06.850749 25745 server_base.cc:1034] running on GCE node
I20260806 09:22:06.851830 25745 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:06.853498 25745 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:06.856974 25745 hybrid_clock.cc:648] HybridClock initialized: now 1786008126856860 us; error 111 us; skew 500 ppm
I20260806 09:22:06.857414 25745 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:06 dist-test-slave-h3ft krb5kdc[25392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:06.861812 25745 init.cc:377] Logged in from keytab as oryx/127.16.86.130@KRBTEST.COM (short username oryx)
I20260806 09:22:06.864138 25745 webserver.cc:460] Webserver started at http://127.16.86.130:41065/ using document root <none> and password file <none>
I20260806 09:22:06.865165 25745 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:06.865340 25745 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:06.865746 25745 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:06.869489 25745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "5cb7873ac0eb467ebce54384aee30caa"
format_stamp: "Formatted at 2026-08-06 09:22:06 on dist-test-slave-h3ft"
server_key: "1a9b48fe0fde2ba84727bac8c65c20c7"
server_key_iv: "9e71f5a581846967c96ee842f2ee9ca4"
server_key_version: "encryptionkey@0"
I20260806 09:22:06.870497 25745 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "5cb7873ac0eb467ebce54384aee30caa"
format_stamp: "Formatted at 2026-08-06 09:22:06 on dist-test-slave-h3ft"
server_key: "1a9b48fe0fde2ba84727bac8c65c20c7"
server_key_iv: "9e71f5a581846967c96ee842f2ee9ca4"
server_key_version: "encryptionkey@0"
I20260806 09:22:06.877369 25745 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:22:06.882400 25766 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:06.884424 25745 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:22:06.884676 25745 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "5cb7873ac0eb467ebce54384aee30caa"
format_stamp: "Formatted at 2026-08-06 09:22:06 on dist-test-slave-h3ft"
server_key: "1a9b48fe0fde2ba84727bac8c65c20c7"
server_key_iv: "9e71f5a581846967c96ee842f2ee9ca4"
server_key_version: "encryptionkey@0"
I20260806 09:22:06.884927 25745 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:06.954671 25745 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:06.960664 25745 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:06.960991 25745 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:06.962200 25745 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:06.964214 25745 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:06.964327 25745 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:06.964498 25745 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:06.964574 25745 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:07.010571 25745 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:33775
I20260806 09:22:07.010634 25922 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:33775 every 8 connection(s)
I20260806 09:22:07.012573 25745 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:22:07.015297 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 25745
I20260806 09:22:07.015542 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:22:07.016283 16730 external_mini_cluster.cc:1428] Setting key 30b162d425f401826d0d90e2ec760aed
Aug 06 09:22:07 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008126, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:07.049073 25927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:07.018382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:34245 (local address 127.16.86.190:43733)
0806 09:22:07.018968 (+ 586us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:07.018972 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:07.021035 (+ 2063us) server_negotiation.cc:407] Connection header received
0806 09:22:07.024280 (+ 3245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:07.024287 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:07.024401 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:07.024543 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:07.027102 (+ 2559us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:07.028446 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:07.031866 (+ 3420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:07.032768 (+ 902us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:07.033029 (+ 261us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:07.038785 (+ 5756us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:07.038797 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:07.038805 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:07.038886 (+ 81us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:07.043681 (+ 4795us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:07.045155 (+ 1474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:07.045165 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:07.045172 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:07.045397 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:07.046153 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:07.046161 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:07.046167 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:07.046676 (+ 509us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:07.046879 (+ 203us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:07.048153 (+ 1274us) server_negotiation.cc:299] Negotiation successful
0806 09:22:07.048491 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":331,"threads_started":1}
I20260806 09:22:07.049829 25928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:07.019484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.130:34245)
0806 09:22:07.020773 (+ 1289us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:07.020841 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:07.023867 (+ 3026us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:07.024913 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:07.024929 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:07.025848 (+ 919us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:07.026790 (+ 942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:07.026820 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:07.028705 (+ 1885us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:07.028714 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:07.031618 (+ 2904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:07.031651 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:07.033287 (+ 1636us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:07.033294 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:07.033516 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:07.034526 (+ 1010us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:07.034560 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:07.038398 (+ 3838us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:07.043939 (+ 5541us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:07.043947 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:07.043962 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:07.044935 (+ 973us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:07.045566 (+ 631us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:07.045570 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:07.045573 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:07.045964 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:07.046857 (+ 893us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:07.046864 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:07.047456 (+ 592us) client_negotiation.cc:769] Sending connection context
0806 09:22:07.047969 (+ 513us) client_negotiation.cc:240] Negotiation successful
0806 09:22:07.048462 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":487,"threads_started":1}
I20260806 09:22:07.051960 25923 heartbeater.cc:347] Connected to a master server at 127.16.86.190:43733
I20260806 09:22:07.052348 25923 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for oryx/127.16.86.131@KRBTEST.COM; defaulting to no policy
I20260806 09:22:07.053282 25923 heartbeater.cc:511] Master 127.16.86.190:43733 requested a full tablet report, sending...
I20260806 09:22:07.055759 25460 ts_manager.cc:194] Registered new tserver with Master: 5cb7873ac0eb467ebce54384aee30caa (127.16.86.130:33775)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.131@KRBTEST.COM" created.
I20260806 09:22:07.056952 25460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:34245
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab.
Entry for principal oryx/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab.
I20260806 09:22:07.100919 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43733
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab
--principal=oryx/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:22:07.318019 25738 heartbeater.cc:503] Master 127.16.86.190:43733 was elected leader, sending a full tablet report...
W20260806 09:22:07.570389 25931 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:07.570854 25931 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:07.571101 25931 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:07.579555 25931 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:07.579771 25931 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:07.579975 25931 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:22:07.591828 25931 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab
--principal=oryx/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43733
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:07.593748 25931 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:07.595592 25931 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:07.613498 25936 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:07.613610 25939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:07.613610 25941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:07.615478 25931 server_base.cc:1034] running on GCE node
I20260806 09:22:07.616400 25931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:07.617692 25931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:07.619061 25931 hybrid_clock.cc:648] HybridClock initialized: now 1786008127618933 us; error 66 us; skew 500 ppm
I20260806 09:22:07.619501 25931 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:07 dist-test-slave-h3ft krb5kdc[25392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:07.623625 25931 init.cc:377] Logged in from keytab as oryx/127.16.86.131@KRBTEST.COM (short username oryx)
I20260806 09:22:07.625499 25931 webserver.cc:460] Webserver started at http://127.16.86.131:35881/ using document root <none> and password file <none>
I20260806 09:22:07.626303 25931 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:07.626425 25931 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:07.626760 25931 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:07.630407 25931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "d8f51643bf134d19843fc65d7fd39502"
format_stamp: "Formatted at 2026-08-06 09:22:07 on dist-test-slave-h3ft"
server_key: "303cfd88d8045929a201c3bda7c20a0a"
server_key_iv: "946dc5743e3d588d184dae32cc717939"
server_key_version: "encryptionkey@0"
I20260806 09:22:07.631364 25931 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "d8f51643bf134d19843fc65d7fd39502"
format_stamp: "Formatted at 2026-08-06 09:22:07 on dist-test-slave-h3ft"
server_key: "303cfd88d8045929a201c3bda7c20a0a"
server_key_iv: "946dc5743e3d588d184dae32cc717939"
server_key_version: "encryptionkey@0"
I20260806 09:22:07.638025 25931 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.004s
I20260806 09:22:07.642370 25953 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:07.644372 25931 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:22:07.644632 25931 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "d8f51643bf134d19843fc65d7fd39502"
format_stamp: "Formatted at 2026-08-06 09:22:07 on dist-test-slave-h3ft"
server_key: "303cfd88d8045929a201c3bda7c20a0a"
server_key_iv: "946dc5743e3d588d184dae32cc717939"
server_key_version: "encryptionkey@0"
I20260806 09:22:07.644860 25931 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:07.677646 25931 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:07.683481 25931 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:07.683862 25931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:07.685340 25931 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:07.687146 25931 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:07.687249 25931 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:07.687469 25931 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:07.687582 25931 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:07.734542 25931 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:36101
I20260806 09:22:07.734720 26109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:36101 every 8 connection(s)
I20260806 09:22:07.736164 25931 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:22:07.746392 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 25931
I20260806 09:22:07.746675 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:22:07.747315 16730 external_mini_cluster.cc:1428] Setting key 1a16d7a2f22e7303882be9978de82020
Aug 06 09:22:07 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008127, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:07.764446 26114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:07.740142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:37659 (local address 127.16.86.190:43733)
0806 09:22:07.740814 (+ 672us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:07.740820 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:07.742408 (+ 1588us) server_negotiation.cc:407] Connection header received
0806 09:22:07.744763 (+ 2355us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:07.744769 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:07.744885 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:07.745027 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:07.747693 (+ 2666us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:07.748732 (+ 1039us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:07.751624 (+ 2892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:07.752264 (+ 640us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:07.752387 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:07.756986 (+ 4599us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:07.756992 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:07.756996 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:07.757040 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:07.760157 (+ 3117us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:07.761141 (+ 984us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:07.761146 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:07.761148 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:07.761310 (+ 162us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:07.761949 (+ 639us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:07.761952 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:07.761954 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:07.762277 (+ 323us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:07.762448 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:07.763652 (+ 1204us) server_negotiation.cc:299] Negotiation successful
0806 09:22:07.763965 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":494,"thread_start_us":327,"threads_started":1}
I20260806 09:22:07.765878 26117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:07.740820 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.131:37659)
0806 09:22:07.741977 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:07.742247 (+ 270us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:07.744402 (+ 2155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:07.745370 (+ 968us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:07.745379 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:07.746017 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:07.747396 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:07.747451 (+ 55us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:07.749050 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:07.749056 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:07.751392 (+ 2336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:07.751418 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:07.752527 (+ 1109us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:07.752533 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:07.752818 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:07.753738 (+ 920us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:07.753767 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:07.756689 (+ 2922us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:07.760341 (+ 3652us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:07.760350 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:07.760372 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:07.760964 (+ 592us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:07.761495 (+ 531us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:07.761502 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:07.761509 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:07.761815 (+ 306us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:07.762455 (+ 640us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:07.762463 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:07.763121 (+ 658us) client_negotiation.cc:769] Sending connection context
0806 09:22:07.763505 (+ 384us) client_negotiation.cc:240] Negotiation successful
0806 09:22:07.764210 (+ 705us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":704,"thread_start_us":419,"threads_started":1}
I20260806 09:22:07.768070 26110 heartbeater.cc:347] Connected to a master server at 127.16.86.190:43733
I20260806 09:22:07.768512 26110 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:07.769491 26110 heartbeater.cc:511] Master 127.16.86.190:43733 requested a full tablet report, sending...
I20260806 09:22:07.771440 25460 ts_manager.cc:194] Registered new tserver with Master: d8f51643bf134d19843fc65d7fd39502 (127.16.86.131:36101)
I20260806 09:22:07.772291 25460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:37659
I20260806 09:22:07.778116 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.86.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.189@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.189.keytab.
Entry for principal oryx/127.16.86.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.189.keytab.
I20260806 09:22:07.852710 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.189:46587
--webserver_interface=127.16.86.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.189.keytab
--principal=oryx/127.16.86.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:43733,127.16.86.189:46587
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:22:08.060776 25923 heartbeater.cc:503] Master 127.16.86.190:43733 was elected leader, sending a full tablet report...
W20260806 09:22:08.239203 26121 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:08.239593 26121 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:08.239768 26121 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:08.247413 26121 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:22:08.247592 26121 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:08.247704 26121 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:08.247813 26121 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:22:08.247900 26121 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:22:08.258353 26121 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:43733,127.16.86.189:46587
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.189.keytab
--principal=oryx/127.16.86.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.189:46587
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.86.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
W20260806 09:22:08.258824 26121 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.16.86.190:43733,127.16.86.189:46587'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260806 09:22:08.260025 26121 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:08.261401 26121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:08.276340 26127 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:08.276270 26129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:08.277958 26131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:08.278759 26121 server_base.cc:1034] running on GCE node
I20260806 09:22:08.280122 26121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:08.282119 26121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:08.283459 26121 hybrid_clock.cc:648] HybridClock initialized: now 1786008128283370 us; error 55 us; skew 500 ppm
I20260806 09:22:08.283967 26121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:08 dist-test-slave-h3ft krb5kdc[25392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008128, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:08.288579 26121 init.cc:377] Logged in from keytab as oryx/127.16.86.189@KRBTEST.COM (short username oryx)
I20260806 09:22:08.290886 26121 webserver.cc:460] Webserver started at http://127.16.86.189:43039/ using document root <none> and password file <none>
I20260806 09:22:08.291879 26121 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:08.292023 26121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:08.292479 26121 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:08.296689 26121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data/instance:
uuid: "fd6d8f8101f447c7a37af25d89cceaf9"
format_stamp: "Formatted at 2026-08-06 09:22:08 on dist-test-slave-h3ft"
server_key: "46b33288f86ff3ca6a878891a8d00a0e"
server_key_iv: "721e7831bdf1d7bf3d0f055f203be31e"
server_key_version: "encryptionkey@0"
I20260806 09:22:08.297775 26121 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/wal/instance:
uuid: "fd6d8f8101f447c7a37af25d89cceaf9"
format_stamp: "Formatted at 2026-08-06 09:22:08 on dist-test-slave-h3ft"
server_key: "46b33288f86ff3ca6a878891a8d00a0e"
server_key_iv: "721e7831bdf1d7bf3d0f055f203be31e"
server_key_version: "encryptionkey@0"
I20260806 09:22:08.305234 26121 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260806 09:22:08.309715 26144 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:08.311829 26121 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:22:08.312107 26121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/wal
uuid: "fd6d8f8101f447c7a37af25d89cceaf9"
format_stamp: "Formatted at 2026-08-06 09:22:08 on dist-test-slave-h3ft"
server_key: "46b33288f86ff3ca6a878891a8d00a0e"
server_key_iv: "721e7831bdf1d7bf3d0f055f203be31e"
server_key_version: "encryptionkey@0"
I20260806 09:22:08.312434 26121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:08.339242 26121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:08.344986 26121 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:08.345391 26121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:08.370983 26121 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.189:46587
I20260806 09:22:08.371091 26223 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.189:46587 every 8 connection(s)
I20260806 09:22:08.373091 26121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
I20260806 09:22:08.377478 26224 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:08.378365 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 26121
I20260806 09:22:08.388238 26224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:22:08 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.86.189@KRBTEST.COM
Aug 06 09:22:08 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008128, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.189@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:08.409343 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.381909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40196 (local address 127.16.86.189:46587)
0806 09:22:08.383044 (+ 1135us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:08.383067 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:08.383147 (+ 80us) server_negotiation.cc:407] Connection header received
0806 09:22:08.384879 (+ 1732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:08.384911 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:08.385473 (+ 562us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:08.385977 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:08.388043 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.390100 (+ 2057us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.392427 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.393632 (+ 1205us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.393838 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.399110 (+ 5272us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:08.399125 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:08.399143 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:08.399200 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:08.403370 (+ 4170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.404704 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.404714 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.404724 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.404987 (+ 263us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.405532 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.405538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.405541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.406244 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:08.406668 (+ 424us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:08.407250 (+ 582us) server_negotiation.cc:299] Negotiation successful
0806 09:22:08.407861 (+ 611us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":799,"thread_start_us":504,"threads_started":1}
I20260806 09:22:08.410585 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.391865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43652 (local address 127.16.86.190:43733)
0806 09:22:08.392430 (+ 565us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:08.392436 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:08.393392 (+ 956us) server_negotiation.cc:407] Connection header received
0806 09:22:08.393599 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:08.393608 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:08.393735 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:08.393904 (+ 169us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:08.395562 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.396507 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.398706 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.399447 (+ 741us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.399628 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.404217 (+ 4589us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:08.404222 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:08.404226 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:08.404267 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:08.407274 (+ 3007us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.408213 (+ 939us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.408221 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.408227 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.408367 (+ 140us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.408797 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.408804 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.408807 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.409174 (+ 367us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:08.409338 (+ 164us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:08.409938 (+ 600us) server_negotiation.cc:299] Negotiation successful
0806 09:22:08.410200 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":290,"threads_started":1}
I20260806 09:22:08.410738 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.392130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:43652)
0806 09:22:08.393157 (+ 1027us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:08.393239 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:08.393421 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:08.394023 (+ 602us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:08.394033 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:08.394932 (+ 899us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:08.395368 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:08.395380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.396684 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.396691 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:08.398431 (+ 1740us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:08.398453 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.399743 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.399750 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:08.399959 (+ 209us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.400893 (+ 934us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:08.400928 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:08.403990 (+ 3062us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:08.407425 (+ 3435us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:08.407438 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:08.407447 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:08.408084 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:08.408494 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:08.408500 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:08.408503 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:08.408674 (+ 171us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:08.409321 (+ 647us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:08.409334 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:08.409587 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 09:22:08.409817 (+ 230us) client_negotiation.cc:240] Negotiation successful
0806 09:22:08.410260 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":734,"thread_start_us":540,"threads_started":1}
I20260806 09:22:08.413014 26224 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:22:08 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008128, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.189@KRBTEST.COM for oryx/127.16.86.189@KRBTEST.COM
I20260806 09:22:08.428746 26232 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.414070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46587 (local address 127.0.0.1:40206)
0806 09:22:08.414341 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:08.414368 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:08.414534 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:08.414941 (+ 407us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:08.414947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:08.415451 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:08.415615 (+ 164us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:08.415624 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.416843 (+ 1219us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.416865 (+ 22us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:08.418030 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:08.418044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.418908 (+ 864us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.418914 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:08.419158 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.419813 (+ 655us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:08.419833 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:08.422562 (+ 2729us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:08.425909 (+ 3347us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:08.425914 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:08.425917 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:08.426422 (+ 505us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:08.426897 (+ 475us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:08.426901 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:08.426904 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:08.427113 (+ 209us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:08.427660 (+ 547us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:08.427664 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:08.427868 (+ 204us) client_negotiation.cc:769] Sending connection context
0806 09:22:08.428067 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 09:22:08.428404 (+ 337us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":98}
I20260806 09:22:08.428960 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.414082 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40206 (local address 127.16.86.189:46587)
0806 09:22:08.414341 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:08.414348 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:08.414459 (+ 111us) server_negotiation.cc:407] Connection header received
0806 09:22:08.414669 (+ 210us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:08.414675 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:08.414752 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:08.414866 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:08.415778 (+ 912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.416673 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.418209 (+ 1536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.418711 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.418875 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.422838 (+ 3963us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:08.422845 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:08.422849 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:08.422890 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:08.425693 (+ 2803us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.426580 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.426584 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.426587 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.426735 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.427222 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.427225 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.427227 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.427546 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:08.427668 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:08.428188 (+ 520us) server_negotiation.cc:299] Negotiation successful
0806 09:22:08.428539 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":65,"server-negotiator.queue_time_us":89}
I20260806 09:22:08.435271 26224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: Bootstrap starting.
I20260806 09:22:08.439436 26224 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:08.440838 26224 log.cc:824] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:08.444720 26224 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: No bootstrap required, opened a new log
I20260806 09:22:08.453981 26224 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } }
I20260806 09:22:08.454493 26224 raft_consensus.cc:374] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:08.454618 26224 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd6d8f8101f447c7a37af25d89cceaf9, State: Initialized, Role: FOLLOWER
I20260806 09:22:08.455492 26224 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } }
I20260806 09:22:08.456730 26234 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } } }
I20260806 09:22:08.457149 26234 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:22:08.458077 26224 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:22:08.475138 26121 master_runner.cc:428] Detected that this master fd6d8f8101f447c7a37af25d89cceaf9 is joining an existing cluster
I20260806 09:22:08.475307 26121 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.86.189:46587
W20260806 09:22:08.485117 26250 catalog_manager.cc:1558] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:22:08.485329 26250 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
WARNING: no policy specified for oryx/127.16.86.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.188@KRBTEST.COM" created.
I20260806 09:22:08.522783 26267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.489581 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:43658)
0806 09:22:08.497155 (+ 7574us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:08.497193 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:08.499069 (+ 1876us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:08.499760 (+ 691us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:08.499765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:08.501189 (+ 1424us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:08.502008 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:08.502027 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.503479 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.503489 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:08.507407 (+ 3918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:08.507425 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.508903 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.508911 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:08.509154 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.510108 (+ 954us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:08.510142 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:08.512058 (+ 1916us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:08.517962 (+ 5904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:08.517972 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:08.517979 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:08.519171 (+ 1192us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:08.519757 (+ 586us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:08.519765 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:08.519772 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:08.520013 (+ 241us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:08.520986 (+ 973us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:08.520995 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:08.521292 (+ 297us) client_negotiation.cc:769] Sending connection context
0806 09:22:08.521477 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 09:22:08.522012 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7251,"mutex_wait_us":68,"thread_start_us":7866,"threads_started":1}
I20260806 09:22:08.524346 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.489029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43658 (local address 127.16.86.190:43733)
0806 09:22:08.489306 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:08.489316 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:08.497408 (+ 8092us) server_negotiation.cc:407] Connection header received
0806 09:22:08.499282 (+ 1874us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:08.499290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:08.499425 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:08.499542 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:08.502238 (+ 2696us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.503296 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.507873 (+ 4577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.508634 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.508800 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.512391 (+ 3591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:08.512403 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:08.512412 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:08.512476 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:08.517621 (+ 5145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.519376 (+ 1755us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.519387 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.519394 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.519587 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.520165 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.520174 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.520181 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.520716 (+ 535us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:08.520906 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:08.523502 (+ 2596us) server_negotiation.cc:299] Negotiation successful
0806 09:22:08.523860 (+ 358us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:22:08.524034 26268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.490135 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46587 (local address 127.0.0.1:40220)
0806 09:22:08.498582 (+ 8447us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:08.498606 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:08.498859 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:08.499760 (+ 901us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:08.499765 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:08.501189 (+ 1424us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:08.502034 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:08.502044 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.507172 (+ 5128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.507179 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:08.508749 (+ 1570us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:08.508770 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.510229 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.510236 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:08.510452 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.511306 (+ 854us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:08.511332 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:08.515949 (+ 4617us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:08.520625 (+ 4676us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:08.520630 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:08.520634 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:08.521215 (+ 581us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:08.521745 (+ 530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:08.521750 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:08.521754 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:08.521943 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:08.522799 (+ 856us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:08.522804 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:08.522966 (+ 162us) client_negotiation.cc:769] Sending connection context
0806 09:22:08.523176 (+ 210us) client_negotiation.cc:240] Negotiation successful
0806 09:22:08.523506 (+ 330us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8301,"mutex_wait_us":107,"spinlock_wait_cycles":43392,"thread_start_us":7988,"threads_started":1}
I20260806 09:22:08.524814 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.489026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40220 (local address 127.16.86.189:46587)
0806 09:22:08.489356 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:08.489365 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:08.498719 (+ 9354us) server_negotiation.cc:407] Connection header received
0806 09:22:08.499297 (+ 578us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:08.499302 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:08.499393 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:08.499478 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:08.502204 (+ 2726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.506786 (+ 4582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.509041 (+ 2255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.509864 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.510007 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.516157 (+ 6150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:08.516164 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:08.516169 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:08.516205 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:08.520459 (+ 4254us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.521433 (+ 974us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.521438 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.521441 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.521619 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.522063 (+ 444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.522076 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.522079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.522534 (+ 455us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:08.522740 (+ 206us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:08.524224 (+ 1484us) server_negotiation.cc:299] Negotiation successful
0806 09:22:08.524446 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
Aug 06 09:22:08 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008125, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.190@KRBTEST.COM for oryx/127.16.86.189@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.188.keytab.
Entry for principal oryx/127.16.86.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.188.keytab.
I20260806 09:22:08.555895 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.188:33963
--webserver_interface=127.16.86.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.188.keytab
--principal=oryx/127.16.86.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:43733,127.16.86.189:46587,127.16.86.188:33963
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:22:08.559252 26272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.533382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46587 (local address 127.0.0.1:53098)
0806 09:22:08.539817 (+ 6435us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:08.539881 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:08.540221 (+ 340us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:08.541051 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:08.541063 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:08.541936 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:08.542213 (+ 277us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:08.542223 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.543533 (+ 1310us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.543540 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:08.544717 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:08.544730 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.545647 (+ 917us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.545654 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:08.545883 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.546719 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:08.546740 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:08.550327 (+ 3587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:08.554976 (+ 4649us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:08.555040 (+ 64us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:08.555051 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:08.555837 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:08.556479 (+ 642us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:08.556487 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:08.556491 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:08.556780 (+ 289us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:08.557714 (+ 934us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:08.557726 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:08.558049 (+ 323us) client_negotiation.cc:769] Sending connection context
0806 09:22:08.558274 (+ 225us) client_negotiation.cc:240] Negotiation successful
0806 09:22:08.558731 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6113,"thread_start_us":5885,"threads_started":1}
I20260806 09:22:08.560470 26229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.533428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53098 (local address 127.16.86.189:46587)
0806 09:22:08.533692 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:08.533704 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:08.540024 (+ 6320us) server_negotiation.cc:407] Connection header received
0806 09:22:08.540522 (+ 498us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:08.540533 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:08.540625 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:08.540790 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:08.542394 (+ 1604us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.543391 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.544907 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.545458 (+ 551us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.545617 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.550638 (+ 5021us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:08.550647 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:08.550653 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:08.550718 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:08.554749 (+ 4031us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.556032 (+ 1283us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.556039 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.556043 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.556245 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.556937 (+ 692us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.556944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.556949 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.557428 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:08.557576 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:08.559836 (+ 2260us) server_negotiation.cc:299] Negotiation successful
0806 09:22:08.560084 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:22:08.563287 25460 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.86.189:46587: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9"
member_type: NON_VOTER
last_known_addr {
host: "127.16.86.189"
port: 46587
}
attrs {
promote: true
}
}
dest_uuid: "4b2fc5f140794fee87c356b29cd9fb6a"
cas_config_opid_index: -1
I20260806 09:22:08.565165 25460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true } }
I20260806 09:22:08.571709 26276 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER fd6d8f8101f447c7a37af25d89cceaf9 (127.16.86.189) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true } } }
I20260806 09:22:08.573290 26184 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:08.577037 26276 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.86.189:46587
I20260806 09:22:08.578033 26184 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 FOLLOWER]: Refusing update from remote peer 4b2fc5f140794fee87c356b29cd9fb6a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260806 09:22:08.578274 26121 master.cc:537] Master@127.16.86.189:46587 shutting down...
I20260806 09:22:08.578435 26276 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true } } }
I20260806 09:22:08.578995 26276 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:08.584370 26274 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 09:22:08.588984 25441 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20260806 09:22:08.598433 26157 tablet.cc:2369] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:22:08.601954 25441 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a -> Peer fd6d8f8101f447c7a37af25d89cceaf9 (127.16.86.189:46587): Couldn't send request to peer fd6d8f8101f447c7a37af25d89cceaf9. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 09:22:08.618654 26121 raft_consensus.cc:2229] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:08.619339 26121 raft_consensus.cc:2258] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:08.619575 26121 tablet_replica.cc:330] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: stopping tablet replica
I20260806 09:22:08.629320 26121 master.cc:552] Master@127.16.86.189:46587 shutdown complete.
I20260806 09:22:08.629489 26121 master_runner.cc:305] Clearing existing system tablet
I20260806 09:22:08.630097 26121 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:08.630218 26121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:08.633919 26121 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.001s sys 0.001s
I20260806 09:22:08.636147 26284 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:08.636694 26121 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:08.637234 26121 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 09:22:08.637389 26121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/wal
uuid: "fd6d8f8101f447c7a37af25d89cceaf9"
format_stamp: "Formatted at 2026-08-06 09:22:08 on dist-test-slave-h3ft"
server_key: "46b33288f86ff3ca6a878891a8d00a0e"
server_key_iv: "721e7831bdf1d7bf3d0f055f203be31e"
server_key_version: "encryptionkey@0"
I20260806 09:22:08.643570 26121 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:08.652032 26121 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 09:22:08.652484 26121 log.cc:1196] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20260806 09:22:08.653146 26121 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: Deleting consensus metadata
I20260806 09:22:08.653666 26121 master_runner.cc:315] Copying system tablet from 127.16.86.190:43733
I20260806 09:22:08.657441 26121 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: tablet copy: Beginning tablet copy session from remote peer at address 127.16.86.190:43733
I20260806 09:22:08.681660 26292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.658929 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:53772)
0806 09:22:08.663626 (+ 4697us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:08.663666 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:08.664019 (+ 353us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:08.664627 (+ 608us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:08.664632 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:08.665304 (+ 672us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:08.665649 (+ 345us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:08.665663 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.667494 (+ 1831us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.667504 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:08.669314 (+ 1810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:08.669336 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.670630 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.670641 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:08.670825 (+ 184us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.671787 (+ 962us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:08.671813 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:08.673071 (+ 1258us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:08.677961 (+ 4890us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:08.677966 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:08.677970 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:08.678586 (+ 616us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:08.679180 (+ 594us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:08.679185 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:08.679189 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:08.679388 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:08.680202 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:08.680209 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:08.680462 (+ 253us) client_negotiation.cc:769] Sending connection context
0806 09:22:08.680711 (+ 249us) client_negotiation.cc:240] Negotiation successful
0806 09:22:08.681103 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4493,"thread_start_us":4053,"threads_started":1}
I20260806 09:22:08.682106 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:08.659431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53772 (local address 127.16.86.190:43733)
0806 09:22:08.659743 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:08.659750 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:08.663832 (+ 4082us) server_negotiation.cc:407] Connection header received
0806 09:22:08.664263 (+ 431us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:08.664271 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:08.664410 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:08.664563 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:08.665857 (+ 1294us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.667191 (+ 1334us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.669526 (+ 2335us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:08.670186 (+ 660us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:08.670291 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:08.673430 (+ 3139us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:08.673439 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:08.673447 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:08.673511 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:08.677748 (+ 4237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.678775 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.678782 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.678787 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.679034 (+ 247us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:08.679528 (+ 494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:08.679534 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:08.679537 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:08.679999 (+ 462us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:08.680171 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:08.680836 (+ 665us) server_negotiation.cc:299] Negotiation successful
0806 09:22:08.681124 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":128}
I20260806 09:22:08.683189 25494 tablet_copy_service.cc:140] P 4b2fc5f140794fee87c356b29cd9fb6a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer fd6d8f8101f447c7a37af25d89cceaf9 ({username='oryx', principal='oryx/127.16.86.189@KRBTEST.COM'} at 127.0.0.1:53772)
I20260806 09:22:08.683661 25494 tablet_copy_service.cc:161] P 4b2fc5f140794fee87c356b29cd9fb6a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer fd6d8f8101f447c7a37af25d89cceaf9 at {username='oryx', principal='oryx/127.16.86.189@KRBTEST.COM'} at 127.0.0.1:53772: session id = fd6d8f8101f447c7a37af25d89cceaf9-00000000000000000000000000000000
I20260806 09:22:08.688875 25494 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 09:22:08.692597 26121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 09:22:08.701159 26121 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: tablet copy: Starting download of 0 data blocks...
I20260806 09:22:08.701877 26121 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: tablet copy: Starting download of 1 WAL segments...
I20260806 09:22:08.707695 26121 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 09:22:08.714411 25494 tablet_copy_service.cc:342] P 4b2fc5f140794fee87c356b29cd9fb6a: Request end of tablet copy session fd6d8f8101f447c7a37af25d89cceaf9-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.86.189@KRBTEST.COM'} at 127.0.0.1:53772
I20260806 09:22:08.714964 25494 tablet_copy_service.cc:433] P 4b2fc5f140794fee87c356b29cd9fb6a: ending tablet copy session fd6d8f8101f447c7a37af25d89cceaf9-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer fd6d8f8101f447c7a37af25d89cceaf9
I20260806 09:22:08.730973 26121 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:08.741449 26294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:08.743840 26293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:08.749274 26297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:08.757256 26121 server_base.cc:1034] running on GCE node
I20260806 09:22:08.757782 26121 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:08.758724 26121 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:08.759974 26121 hybrid_clock.cc:648] HybridClock initialized: now 1786008128759888 us; error 59 us; skew 500 ppm
I20260806 09:22:08.760254 26121 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:08 dist-test-slave-h3ft krb5kdc[25392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008128, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:08.762877 26121 init.cc:377] Logged in from keytab as oryx/127.16.86.189@KRBTEST.COM (short username oryx)
I20260806 09:22:08.768635 26121 webserver.cc:460] Webserver started at http://127.16.86.189:39075/ using document root <none> and password file <none>
I20260806 09:22:08.769163 26121 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:08.769269 26121 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:08.773084 26121 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260806 09:22:08.775697 26307 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:08.775885 26110 heartbeater.cc:503] Master 127.16.86.190:43733 was elected leader, sending a full tablet report...
I20260806 09:22:08.777009 26121 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:22:08.777146 26121 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/wal
uuid: "fd6d8f8101f447c7a37af25d89cceaf9"
format_stamp: "Formatted at 2026-08-06 09:22:08 on dist-test-slave-h3ft"
server_key: "46b33288f86ff3ca6a878891a8d00a0e"
server_key_iv: "721e7831bdf1d7bf3d0f055f203be31e"
server_key_version: "encryptionkey@0"
I20260806 09:22:08.777323 26121 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:08.822734 26121 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:08.827873 26121 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:08.922415 26121 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.189:46587
I20260806 09:22:08.922638 26376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.189:46587 every 8 connection(s)
I20260806 09:22:08.924074 26121 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
I20260806 09:22:08.930497 26377 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:22:08.935141 26377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: Bootstrap starting.
I20260806 09:22:08.956770 26377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:22:08.957509 26377 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: Bootstrap complete.
I20260806 09:22:08.958837 26377 raft_consensus.cc:348] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true } }
I20260806 09:22:08.959209 26377 raft_consensus.cc:729] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: fd6d8f8101f447c7a37af25d89cceaf9, State: Initialized, Role: LEARNER
I20260806 09:22:08.959599 26377 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true } }
I20260806 09:22:08.961182 26377 sys_catalog.cc:564] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:22:08.960964 26382 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true } } }
I20260806 09:22:08.961735 26382 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:22:08.973404 26397 catalog_manager.cc:1259] Loaded cluster ID: 4c0afa372dac4077b1e53622c7454eba
I20260806 09:22:08.973634 26397 catalog_manager.cc:1552] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: loading cluster ID for follower catalog manager: success
I20260806 09:22:08.978250 26397 catalog_manager.cc:1574] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: acquiring CA information for follower catalog manager: success
I20260806 09:22:08.981508 26397 catalog_manager.cc:1602] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 09:22:09.003417 26277 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:09.003813 26277 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:09.003980 26277 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:09.012897 26277 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:22:09.013082 26277 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:09.013175 26277 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:09.013262 26277 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:22:09.013347 26277 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:22:09.025048 26277 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:43733,127.16.86.189:46587,127.16.86.188:33963
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.188.keytab
--principal=oryx/127.16.86.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.188:33963
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.86.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:09.026932 26277 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:09.028534 26277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:09.044211 26403 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:09.044167 26405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:09.044521 26410 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:09.045709 26277 server_base.cc:1034] running on GCE node
I20260806 09:22:09.047077 26277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:09.049307 26277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:09.050581 26277 hybrid_clock.cc:648] HybridClock initialized: now 1786008129050500 us; error 74 us; skew 500 ppm
I20260806 09:22:09.051163 26277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:09 dist-test-slave-h3ft krb5kdc[25392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008129, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:09.055919 26277 init.cc:377] Logged in from keytab as oryx/127.16.86.188@KRBTEST.COM (short username oryx)
I20260806 09:22:09.058331 26277 webserver.cc:460] Webserver started at http://127.16.86.188:41179/ using document root <none> and password file <none>
I20260806 09:22:09.059351 26277 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:09.059506 26277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:09.059909 26277 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:09.064625 26277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data/instance:
uuid: "dee9589594ac4444bff7783dc8b59f3b"
format_stamp: "Formatted at 2026-08-06 09:22:09 on dist-test-slave-h3ft"
server_key: "3742fc5e6f21e98c9af02531bd7d186c"
server_key_iv: "340e483fd4ee67c6ed6d890132c4b4cc"
server_key_version: "encryptionkey@0"
I20260806 09:22:09.065696 26277 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/wal/instance:
uuid: "dee9589594ac4444bff7783dc8b59f3b"
format_stamp: "Formatted at 2026-08-06 09:22:09 on dist-test-slave-h3ft"
server_key: "3742fc5e6f21e98c9af02531bd7d186c"
server_key_iv: "340e483fd4ee67c6ed6d890132c4b4cc"
server_key_version: "encryptionkey@0"
I20260806 09:22:09.073757 26277 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.001s sys 0.008s
I20260806 09:22:09.079393 26420 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:09.081775 26277 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 09:22:09.082041 26277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/wal
uuid: "dee9589594ac4444bff7783dc8b59f3b"
format_stamp: "Formatted at 2026-08-06 09:22:09 on dist-test-slave-h3ft"
server_key: "3742fc5e6f21e98c9af02531bd7d186c"
server_key_iv: "340e483fd4ee67c6ed6d890132c4b4cc"
server_key_version: "encryptionkey@0"
I20260806 09:22:09.082309 26277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:09.114387 26277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:09.118860 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.098182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46587 (local address 127.0.0.1:53114)
0806 09:22:09.111189 (+ 13007us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:09.111227 (+ 38us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:09.111494 (+ 267us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:09.112047 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:09.112052 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:09.112079 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:09.112320 (+ 241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.112329 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.114216 (+ 1887us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.114222 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.116130 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.116148 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.117950 (+ 1802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.117956 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.118109 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.118167 (+ 58us) client_negotiation.cc:769] Sending connection context
0806 09:22:09.118264 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 09:22:09.118403 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12835,"thread_start_us":12661,"threads_started":1}
I20260806 09:22:09.118917 26430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.098550 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53114 (local address 127.16.86.189:46587)
0806 09:22:09.110787 (+ 12237us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:09.110794 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:09.111375 (+ 581us) server_negotiation.cc:407] Connection header received
0806 09:22:09.111674 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:09.111680 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:09.111795 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:09.111965 (+ 170us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:09.112491 (+ 526us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.113991 (+ 1500us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.116340 (+ 2349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.117699 (+ 1359us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.117883 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.118144 (+ 261us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:09.118334 (+ 190us) server_negotiation.cc:299] Negotiation successful
0806 09:22:09.118519 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12046,"thread_start_us":11856,"threads_started":1}
I20260806 09:22:09.120527 26277 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:09.120764 26342 raft_consensus.cc:1203] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20260806 09:22:09.120939 26277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:09.121466 26382 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b2fc5f140794fee87c356b29cd9fb6a. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true } } }
I20260806 09:22:09.121932 26382 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:22:09.144484 26277 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.188:33963
I20260806 09:22:09.144675 26500 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.188:33963 every 8 connection(s)
I20260806 09:22:09.146123 26277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
I20260806 09:22:09.152004 26503 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:09.156577 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 26277
I20260806 09:22:09.162217 26503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:22:09 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.86.188@KRBTEST.COM
Aug 06 09:22:09 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008129, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.188@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:09.188931 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.167151 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53784 (local address 127.16.86.190:43733)
0806 09:22:09.167365 (+ 214us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:09.167370 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:09.168365 (+ 995us) server_negotiation.cc:407] Connection header received
0806 09:22:09.168671 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:09.168678 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:09.168763 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:09.168896 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:09.170815 (+ 1919us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.172183 (+ 1368us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.174753 (+ 2570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.175301 (+ 548us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.175420 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.180140 (+ 4720us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:09.180149 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:09.180157 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:09.180202 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:09.184072 (+ 3870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.185336 (+ 1264us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.185342 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.185346 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.185559 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.186175 (+ 616us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.186180 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.186183 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.186608 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:09.186786 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:09.188194 (+ 1408us) server_negotiation.cc:299] Negotiation successful
0806 09:22:09.188496 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:22:09.190478 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.159688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51074 (local address 127.16.86.188:33963)
0806 09:22:09.160631 (+ 943us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:09.160645 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:09.160692 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:22:09.163130 (+ 2438us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:09.163158 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:09.163666 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:09.164093 (+ 427us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:09.166740 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.169155 (+ 2415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.171312 (+ 2157us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.172473 (+ 1161us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.172655 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.179593 (+ 6938us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:09.179603 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:09.179618 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:09.179668 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:09.184153 (+ 4485us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.185364 (+ 1211us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.185374 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.185377 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.185603 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.186193 (+ 590us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.186198 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.186201 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.186960 (+ 759us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:09.187324 (+ 364us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:09.187856 (+ 532us) server_negotiation.cc:299] Negotiation successful
0806 09:22:09.188854 (+ 998us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":635,"server-negotiator.queue_time_us":703,"thread_start_us":446,"threads_started":1}
I20260806 09:22:09.190518 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.166984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:53784)
0806 09:22:09.168192 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:09.168238 (+ 46us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:09.168506 (+ 268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:09.169014 (+ 508us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:09.169023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:09.170161 (+ 1138us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:09.170594 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.170609 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.172408 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.172413 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.174565 (+ 2152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.174583 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.175618 (+ 1035us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.175628 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.175885 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.176685 (+ 800us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:09.176734 (+ 49us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:09.179895 (+ 3161us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:09.184307 (+ 4412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.184323 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.184340 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.185152 (+ 812us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.185752 (+ 600us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.185758 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.185762 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.186015 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.186821 (+ 806us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:09.186828 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:09.187224 (+ 396us) client_negotiation.cc:769] Sending connection context
0806 09:22:09.187683 (+ 459us) client_negotiation.cc:240] Negotiation successful
0806 09:22:09.188474 (+ 791us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":924,"thread_start_us":525,"threads_started":1}
I20260806 09:22:09.192878 26503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:22:09 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008129, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.188@KRBTEST.COM for oryx/127.16.86.189@KRBTEST.COM
I20260806 09:22:09.208889 26430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.194118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53130 (local address 127.16.86.189:46587)
0806 09:22:09.194374 (+ 256us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:09.194381 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:09.194413 (+ 32us) server_negotiation.cc:407] Connection header received
0806 09:22:09.194484 (+ 71us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:09.194488 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:09.194560 (+ 72us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:09.194680 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:09.195590 (+ 910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.196391 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.197675 (+ 1284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.198116 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.198257 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.202324 (+ 4067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:09.202332 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:09.202335 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:09.202382 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:09.205550 (+ 3168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.206519 (+ 969us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.206523 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.206526 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.206742 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.207276 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.207280 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.207282 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.207656 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:09.207783 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:09.208351 (+ 568us) server_negotiation.cc:299] Negotiation successful
0806 09:22:09.208626 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:22:09.208901 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.193852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46587 (local address 127.0.0.1:53130)
0806 09:22:09.194113 (+ 261us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:09.194138 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:09.194298 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:09.194712 (+ 414us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:09.194716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:09.195240 (+ 524us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:09.195407 (+ 167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.195416 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.196535 (+ 1119us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.196542 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.197531 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.197542 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.198282 (+ 740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.198288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.198511 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.199346 (+ 835us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:09.199370 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:09.202043 (+ 2673us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:09.205754 (+ 3711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.205760 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.205764 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.206335 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.206894 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.206900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.206902 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.207155 (+ 253us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.207807 (+ 652us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:09.207812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:09.208076 (+ 264us) client_negotiation.cc:769] Sending connection context
0806 09:22:09.208218 (+ 142us) client_negotiation.cc:240] Negotiation successful
0806 09:22:09.208562 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":90}
I20260806 09:22:09.210260 26503 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:22:09 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008129, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.188@KRBTEST.COM for oryx/127.16.86.188@KRBTEST.COM
I20260806 09:22:09.226114 26513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.211189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:33963 (local address 127.0.0.1:51076)
0806 09:22:09.211498 (+ 309us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:09.211517 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:09.211613 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:09.212061 (+ 448us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:09.212066 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:09.212533 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:09.212776 (+ 243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.212786 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.214026 (+ 1240us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.214030 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.215124 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.215138 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.215892 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.215897 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.216100 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.217007 (+ 907us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:09.217030 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:09.219592 (+ 2562us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:09.223199 (+ 3607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.223204 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.223207 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.223709 (+ 502us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.224178 (+ 469us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.224182 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.224184 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.224367 (+ 183us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.225034 (+ 667us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:09.225047 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:09.225223 (+ 176us) client_negotiation.cc:769] Sending connection context
0806 09:22:09.225351 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 09:22:09.225640 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 09:22:09.226145 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.211474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51076 (local address 127.16.86.188:33963)
0806 09:22:09.211722 (+ 248us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:09.211727 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:09.211746 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:22:09.211803 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:09.211807 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:09.211873 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:09.211976 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:09.213001 (+ 1025us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.213851 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.215277 (+ 1426us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.215726 (+ 449us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.215842 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.219837 (+ 3995us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:09.219842 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:09.219845 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:09.219883 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:09.222905 (+ 3022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.223873 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.223878 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.223880 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.224030 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.224548 (+ 518us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.224553 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.224555 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.224892 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:09.225035 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:09.225516 (+ 481us) server_negotiation.cc:299] Negotiation successful
0806 09:22:09.225809 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":57,"server-negotiator.queue_time_us":71}
I20260806 09:22:09.232978 26503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: Bootstrap starting.
I20260806 09:22:09.237628 26503 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:09.239207 26503 log.cc:824] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:09.243323 26503 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: No bootstrap required, opened a new log
I20260806 09:22:09.253432 26503 raft_consensus.cc:348] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } }
I20260806 09:22:09.253958 26503 raft_consensus.cc:374] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:09.254145 26503 raft_consensus.cc:729] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dee9589594ac4444bff7783dc8b59f3b, State: Initialized, Role: FOLLOWER
I20260806 09:22:09.254910 26503 consensus_queue.cc:260] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } }
I20260806 09:22:09.256240 26517 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } } }
I20260806 09:22:09.256698 26517 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:22:09.257647 26503 sys_catalog.cc:564] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: configured and running, proceeding with master startup.
W20260806 09:22:09.270278 26534 catalog_manager.cc:1558] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:22:09.270494 26534 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:22:09.276976 26277 master_runner.cc:428] Detected that this master dee9589594ac4444bff7783dc8b59f3b is joining an existing cluster
I20260806 09:22:09.277196 26277 master_runner.cc:432] Initiating AddMaster RPC to add 127.16.86.188:33963
I20260806 09:22:09.300201 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 25558
I20260806 09:22:09.308580 26544 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.282302 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:53800)
0806 09:22:09.283788 (+ 1486us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:09.283814 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:09.284102 (+ 288us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:09.284805 (+ 703us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:09.284813 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:09.285822 (+ 1009us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:09.286409 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.286426 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.288537 (+ 2111us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.288545 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.292121 (+ 3576us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.292160 (+ 39us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.295147 (+ 2987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.295159 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.295404 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.297023 (+ 1619us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:09.297052 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:09.298590 (+ 1538us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:09.303746 (+ 5156us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.303755 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.303764 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.304690 (+ 926us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.305311 (+ 621us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.305318 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.305324 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.305650 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.306587 (+ 937us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:09.306595 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:09.306951 (+ 356us) client_negotiation.cc:769] Sending connection context
0806 09:22:09.307218 (+ 267us) client_negotiation.cc:240] Negotiation successful
0806 09:22:09.307805 (+ 587us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1267,"thread_start_us":9257,"threads_started":1}
I20260806 09:22:09.309837 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.282840 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53800 (local address 127.16.86.190:43733)
0806 09:22:09.283105 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:09.283114 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:09.283929 (+ 815us) server_negotiation.cc:407] Connection header received
0806 09:22:09.284309 (+ 380us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:09.284317 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:09.284428 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:09.284520 (+ 92us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:09.286631 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.288199 (+ 1568us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.293336 (+ 5137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.294151 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.294323 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.298879 (+ 4556us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:09.298888 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:09.298893 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:09.298948 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:09.303505 (+ 4557us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.304891 (+ 1386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.304901 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.304907 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.305129 (+ 222us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.305819 (+ 690us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.305827 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.305833 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.306319 (+ 486us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:09.306502 (+ 183us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:09.308994 (+ 2492us) server_negotiation.cc:299] Negotiation successful
0806 09:22:09.309283 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 09:22:09.310513 26517 raft_consensus.cc:482] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:22:09.310932 26517 raft_consensus.cc:504] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } }
I20260806 09:22:09.313221 26517 leader_election.cc:288] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b2fc5f140794fee87c356b29cd9fb6a (127.16.86.190:43733), fd6d8f8101f447c7a37af25d89cceaf9 (127.16.86.189:46587)
I20260806 09:22:09.315305 25478 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dee9589594ac4444bff7783dc8b59f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" is_pre_election: true
I20260806 09:22:09.315303 26342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dee9589594ac4444bff7783dc8b59f3b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" is_pre_election: true
I20260806 09:22:09.316685 26421 leader_election.cc:302] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: dee9589594ac4444bff7783dc8b59f3b; no voters: 4b2fc5f140794fee87c356b29cd9fb6a, fd6d8f8101f447c7a37af25d89cceaf9
I20260806 09:22:09.319224 26517 raft_consensus.cc:2726] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 09:22:09.324163 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.284017 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51084 (local address 127.16.86.188:33963)
0806 09:22:09.284251 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:09.284258 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:09.293106 (+ 8848us) server_negotiation.cc:407] Connection header received
0806 09:22:09.293272 (+ 166us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:09.293277 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:09.293347 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:09.293406 (+ 59us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:09.296918 (+ 3512us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.298158 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.301778 (+ 3620us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.302575 (+ 797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.302686 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.307465 (+ 4779us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:09.307472 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:09.307477 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:09.307514 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:09.312005 (+ 4491us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.318258 (+ 6253us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.318264 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.318268 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.318474 (+ 206us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.318924 (+ 450us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.318929 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.318932 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.319464 (+ 532us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:09.319639 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:09.322258 (+ 2619us) server_negotiation.cc:299] Negotiation successful
0806 09:22:09.322650 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":79}
I20260806 09:22:09.332644 26546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.284383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:33963 (local address 127.0.0.1:51084)
0806 09:22:09.292964 (+ 8581us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:09.292984 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:09.293135 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:09.293590 (+ 455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:09.293595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:09.296291 (+ 2696us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:09.296717 (+ 426us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.296731 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.299887 (+ 3156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.299895 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.301598 (+ 1703us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.301615 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.303067 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.303075 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.303266 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.304082 (+ 816us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:09.304107 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:09.307241 (+ 3134us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:09.312212 (+ 4971us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.312223 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.312229 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.312788 (+ 559us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.318628 (+ 5840us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.318632 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.318636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.318795 (+ 159us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.320117 (+ 1322us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:09.320122 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:09.320321 (+ 199us) client_negotiation.cc:769] Sending connection context
0806 09:22:09.331532 (+ 11211us) client_negotiation.cc:240] Negotiation successful
0806 09:22:09.331989 (+ 457us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8187,"thread_start_us":7995,"threads_started":1}
Aug 06 09:22:09 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008125, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.190@KRBTEST.COM for oryx/127.16.86.188@KRBTEST.COM
I20260806 09:22:09.334127 26430 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.282497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53134 (local address 127.16.86.189:46587)
0806 09:22:09.282709 (+ 212us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:09.282716 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:09.292568 (+ 9852us) server_negotiation.cc:407] Connection header received
0806 09:22:09.293181 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:09.293190 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:09.293285 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:09.293359 (+ 74us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:09.296153 (+ 2794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.297069 (+ 916us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.299209 (+ 2140us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.299886 (+ 677us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.299990 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.313077 (+ 13087us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:09.313088 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:09.313093 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:09.313151 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:09.317243 (+ 4092us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.319742 (+ 2499us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.319749 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.319755 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.319945 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.320921 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.320927 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.320932 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.321440 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:09.321595 (+ 155us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:09.333244 (+ 11649us) server_negotiation.cc:299] Negotiation successful
0806 09:22:09.333639 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:22:09.335497 26541 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.282657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46587 (local address 127.0.0.1:53134)
0806 09:22:09.292421 (+ 9764us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:09.292447 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:09.292930 (+ 483us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:09.293590 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:09.293595 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:09.295474 (+ 1879us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:09.295950 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.295965 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.297294 (+ 1329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.297300 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.298983 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.299031 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.300353 (+ 1322us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.300360 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.300573 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.304688 (+ 4115us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:09.304711 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:09.306296 (+ 1585us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:09.317495 (+ 11199us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.317501 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.317506 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.318092 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.320553 (+ 2461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.320559 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.320562 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.320776 (+ 214us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.321720 (+ 944us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:09.321724 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:09.321953 (+ 229us) client_negotiation.cc:769] Sending connection context
0806 09:22:09.334479 (+ 12526us) client_negotiation.cc:240] Negotiation successful
0806 09:22:09.334921 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9299,"mutex_wait_us":3,"thread_start_us":644,"threads_started":1}
Aug 06 09:22:09 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.86.129@KRBTEST.COM
I20260806 09:22:09.346212 26429 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.322707 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:33963 (local address 127.0.0.1:51090)
0806 09:22:09.322966 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:09.322992 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:09.323176 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:09.325199 (+ 2023us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:09.325205 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:09.325865 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:09.326126 (+ 261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.326142 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.327941 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.327947 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.329446 (+ 1499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.329464 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.330649 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.330656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.330872 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.331743 (+ 871us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:09.331776 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:09.335126 (+ 3350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:09.342432 (+ 7306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.342441 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.342446 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.343187 (+ 741us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.343797 (+ 610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.343807 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.343810 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.344057 (+ 247us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.344956 (+ 899us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:09.344962 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:09.345234 (+ 272us) client_negotiation.cc:769] Sending connection context
0806 09:22:09.345409 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 09:22:09.345790 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:22:09.347635 26507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.323078 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51090 (local address 127.16.86.188:33963)
0806 09:22:09.324451 (+ 1373us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:09.324456 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:09.324474 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:22:09.324586 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:09.324592 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:09.324717 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:09.326295 (+ 1578us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:09.326512 (+ 217us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.327768 (+ 1256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.329656 (+ 1888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.330455 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.331351 (+ 896us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.337394 (+ 6043us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:09.337402 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:09.337408 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:09.337457 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:09.342048 (+ 4591us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.343402 (+ 1354us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.343408 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.343412 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.343579 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.344253 (+ 674us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.344258 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.344262 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.344744 (+ 482us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:09.346745 (+ 2001us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:09.346894 (+ 149us) server_negotiation.cc:299] Negotiation successful
0806 09:22:09.347236 (+ 342us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1263,"thread_start_us":20109,"threads_started":1}
I20260806 09:22:09.349375 26550 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.302196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52374 (local address 127.16.86.129:45867)
0806 09:22:09.320427 (+ 18231us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:09.320436 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:09.320469 (+ 33us) server_negotiation.cc:407] Connection header received
0806 09:22:09.320552 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:09.320563 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:09.320886 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:09.323483 (+ 2597us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:09.323884 (+ 401us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.327205 (+ 3321us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.329968 (+ 2763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.331219 (+ 1251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.339061 (+ 7842us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.339495 (+ 434us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:09.339518 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:09.339548 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:09.339607 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:09.345379 (+ 5772us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.346174 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.346184 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.346188 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.346403 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.346861 (+ 458us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.346864 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.346866 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.347295 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:09.347636 (+ 341us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:09.348495 (+ 859us) server_negotiation.cc:299] Negotiation successful
0806 09:22:09.348879 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":18010,"thread_start_us":18828,"threads_started":1}
I20260806 09:22:09.349315 25460 catalog_manager.cc:6979] Initiating ChangeConfig request to add master 127.16.86.188:33963: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b"
member_type: NON_VOTER
last_known_addr {
host: "127.16.86.188"
port: 33963
}
attrs {
promote: true
}
}
dest_uuid: "4b2fc5f140794fee87c356b29cd9fb6a"
cas_config_opid_index: 5
I20260806 09:22:09.350514 25460 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } }
I20260806 09:22:09.351456 25733 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:09.352661 26428 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER dee9589594ac4444bff7783dc8b59f3b (127.16.86.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } } }
I20260806 09:22:09.358822 26342 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 LEARNER]: Refusing update from remote peer 4b2fc5f140794fee87c356b29cd9fb6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260806 09:22:09.360302 26428 catalog_manager.cc:6983] Successfully completed master ChangeConfig request to add master 127.16.86.188:33963
I20260806 09:22:09.361562 26277 master.cc:537] Master@127.16.86.188:33963 shutting down...
W20260806 09:22:09.367152 25441 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a -> Peer dee9589594ac4444bff7783dc8b59f3b (127.16.86.188:33963): Couldn't send request to peer dee9589594ac4444bff7783dc8b59f3b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 09:22:09.373368 26556 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260806 09:22:09.374156 26432 tablet.cc:2369] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:22:09.378015 26342 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER dee9589594ac4444bff7783dc8b59f3b (127.16.86.188) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } } }
I20260806 09:22:09.384254 26382 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } } }
I20260806 09:22:09.384805 26382 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: This master's current role is: LEARNER
I20260806 09:22:09.386904 26428 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a: attempting to promote NON_VOTER fd6d8f8101f447c7a37af25d89cceaf9 to VOTER
I20260806 09:22:09.386970 26563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: NON_VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: true } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } } }
I20260806 09:22:09.387742 26563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:09.387964 26428 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } }
W20260806 09:22:09.391901 25441 consensus_peers.cc:487] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a -> Peer dee9589594ac4444bff7783dc8b59f3b (127.16.86.188:33963): Couldn't send request to peer dee9589594ac4444bff7783dc8b59f3b. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
I20260806 09:22:09.393199 26342 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 LEARNER]: Refusing update from remote peer 4b2fc5f140794fee87c356b29cd9fb6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20260806 09:22:09.393918 26564 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260806 09:22:09.397462 26556 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fd6d8f8101f447c7a37af25d89cceaf9 (127.16.86.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } } }
I20260806 09:22:09.399370 26342 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, fd6d8f8101f447c7a37af25d89cceaf9 (127.16.86.189) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } } }
I20260806 09:22:09.403149 26564 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } } }
I20260806 09:22:09.403295 26556 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } } }
I20260806 09:22:09.403631 26564 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:09.403743 26556 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:09.405011 26382 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } } }
I20260806 09:22:09.405331 26382 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:22:09.408176 26277 raft_consensus.cc:2229] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 0 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:09.408775 26277 raft_consensus.cc:2258] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 0 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:09.409013 26277 tablet_replica.cc:330] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: stopping tablet replica
I20260806 09:22:09.418083 26277 master.cc:552] Master@127.16.86.188:33963 shutdown complete.
I20260806 09:22:09.418208 26277 master_runner.cc:305] Clearing existing system tablet
I20260806 09:22:09.418700 26277 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:09.418833 26277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:09.422492 26277 fs_manager.cc:709] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260806 09:22:09.424602 26572 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:09.425040 26277 fs_report.cc:383] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:09.425578 26277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260806 09:22:09.425753 26277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/wal
uuid: "dee9589594ac4444bff7783dc8b59f3b"
format_stamp: "Formatted at 2026-08-06 09:22:09 on dist-test-slave-h3ft"
server_key: "3742fc5e6f21e98c9af02531bd7d186c"
server_key_iv: "340e483fd4ee67c6ed6d890132c4b4cc"
server_key_version: "encryptionkey@0"
I20260806 09:22:09.431689 26277 ts_tablet_manager.cc:1905] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:09.439997 26277 ts_tablet_manager.cc:1918] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260806 09:22:09.440241 26277 log.cc:1196] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
I20260806 09:22:09.440869 26277 ts_tablet_manager.cc:1939] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: Deleting consensus metadata
I20260806 09:22:09.441170 26277 master_runner.cc:315] Copying system tablet from 127.16.86.190:43733
I20260806 09:22:09.445001 26277 tablet_copy_client.cc:316] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: tablet copy: Beginning tablet copy session from remote peer at address 127.16.86.190:43733
I20260806 09:22:09.469411 26581 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.446026 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:53810)
0806 09:22:09.451578 (+ 5552us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:09.451608 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:09.451842 (+ 234us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:09.452374 (+ 532us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:09.452379 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:09.453108 (+ 729us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:09.453483 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.453498 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.455147 (+ 1649us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.455153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.456826 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.456843 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.458035 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.458042 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:09.458244 (+ 202us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.459244 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:09.459271 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:09.460579 (+ 1308us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:09.465857 (+ 5278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.465865 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.465870 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.466546 (+ 676us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.467060 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:09.467080 (+ 20us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:09.467085 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:09.467273 (+ 188us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:09.468068 (+ 795us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:09.468086 (+ 18us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:09.468297 (+ 211us) client_negotiation.cc:769] Sending connection context
0806 09:22:09.468513 (+ 216us) client_negotiation.cc:240] Negotiation successful
0806 09:22:09.468878 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5354,"thread_start_us":5175,"threads_started":1}
I20260806 09:22:09.470605 26230 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.446121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53810 (local address 127.16.86.190:43733)
0806 09:22:09.446439 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:09.446446 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:09.451787 (+ 5341us) server_negotiation.cc:407] Connection header received
0806 09:22:09.452021 (+ 234us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:09.452028 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:09.452146 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:09.452305 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:09.453688 (+ 1383us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.454941 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.457026 (+ 2085us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.457810 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.457960 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:09.460854 (+ 2894us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:09.460862 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:09.460867 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:09.460920 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:09.465628 (+ 4708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.466717 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.466723 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.466728 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.466891 (+ 163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:09.467415 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:09.467421 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:09.467425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:09.467831 (+ 406us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:09.467989 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:09.469938 (+ 1949us) server_negotiation.cc:299] Negotiation successful
0806 09:22:09.470212 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":147}
I20260806 09:22:09.471753 25494 tablet_copy_service.cc:140] P 4b2fc5f140794fee87c356b29cd9fb6a: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer dee9589594ac4444bff7783dc8b59f3b ({username='oryx', principal='oryx/127.16.86.188@KRBTEST.COM'} at 127.0.0.1:53810)
I20260806 09:22:09.471942 25494 tablet_copy_service.cc:161] P 4b2fc5f140794fee87c356b29cd9fb6a: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer dee9589594ac4444bff7783dc8b59f3b at {username='oryx', principal='oryx/127.16.86.188@KRBTEST.COM'} at 127.0.0.1:53810: session id = dee9589594ac4444bff7783dc8b59f3b-00000000000000000000000000000000
I20260806 09:22:09.476456 25494 tablet_copy_source_session.cc:205] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a: Tablet Copy: opened 0 blocks and 1 log segments
I20260806 09:22:09.479295 26277 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260806 09:22:09.487133 26277 tablet_copy_client.cc:791] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: tablet copy: Starting download of 0 data blocks...
I20260806 09:22:09.487586 26277 tablet_copy_client.cc:655] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: tablet copy: Starting download of 1 WAL segments...
I20260806 09:22:09.490947 26277 tablet_copy_client.cc:531] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260806 09:22:09.496210 25494 tablet_copy_service.cc:342] P 4b2fc5f140794fee87c356b29cd9fb6a: Request end of tablet copy session dee9589594ac4444bff7783dc8b59f3b-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.16.86.188@KRBTEST.COM'} at 127.0.0.1:53810
I20260806 09:22:09.496371 25494 tablet_copy_service.cc:433] P 4b2fc5f140794fee87c356b29cd9fb6a: ending tablet copy session dee9589594ac4444bff7783dc8b59f3b-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer dee9589594ac4444bff7783dc8b59f3b
I20260806 09:22:09.511678 26277 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:09.515180 26582 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:09.515684 26583 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:09.527997 26277 server_base.cc:1034] running on GCE node
W20260806 09:22:09.528434 26586 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:09.529482 26277 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:09.530411 26277 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:09.531446 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 25558
I20260806 09:22:09.531787 26277 hybrid_clock.cc:648] HybridClock initialized: now 1786008129531736 us; error 48 us; skew 500 ppm
I20260806 09:22:09.532143 26277 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:09 dist-test-slave-h3ft krb5kdc[25392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008129, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:09.537273 26277 init.cc:377] Logged in from keytab as oryx/127.16.86.188@KRBTEST.COM (short username oryx)
I20260806 09:22:09.545912 26277 webserver.cc:460] Webserver started at http://127.16.86.188:46347/ using document root <none> and password file <none>
I20260806 09:22:09.546480 26277 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:09.546635 26277 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:09.550789 26277 fs_manager.cc:709] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260806 09:22:09.553485 26598 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:09.554148 26277 fs_manager.cc:725] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20260806 09:22:09.554316 26277 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/wal
uuid: "dee9589594ac4444bff7783dc8b59f3b"
format_stamp: "Formatted at 2026-08-06 09:22:09 on dist-test-slave-h3ft"
server_key: "3742fc5e6f21e98c9af02531bd7d186c"
server_key_iv: "340e483fd4ee67c6ed6d890132c4b4cc"
server_key_version: "encryptionkey@0"
I20260806 09:22:09.554544 26277 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:09.574440 26277 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:09.575827 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:45867
--local_ip_for_outbound_sockets=127.16.86.129
--tserver_master_addrs=127.16.86.190:43733
--webserver_port=41761
--webserver_interface=127.16.86.129
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab
--principal=oryx/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.86.190:43733,127.16.86.189:46587
--tserver_master_addrs=127.16.86.190:43733,127.16.86.189:46587,127.16.86.188:33963 with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:22:09.579249 26277 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:09.608433 26277 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.188:33963
I20260806 09:22:09.608516 26675 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.188:33963 every 8 connection(s)
I20260806 09:22:09.610001 26277 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
I20260806 09:22:09.616472 26676 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:22:09.620820 26676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: Bootstrap starting.
I20260806 09:22:09.642719 26676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:22:09.643450 26676 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: Bootstrap complete.
I20260806 09:22:09.644718 26676 raft_consensus.cc:348] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } }
I20260806 09:22:09.645056 26676 raft_consensus.cc:729] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: dee9589594ac4444bff7783dc8b59f3b, State: Initialized, Role: LEARNER
I20260806 09:22:09.645362 26676 consensus_queue.cc:260] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } }
I20260806 09:22:09.646749 26676 sys_catalog.cc:564] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:22:09.646552 26681 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } } }
I20260806 09:22:09.647397 26681 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: This master's current role is: LEARNER
I20260806 09:22:09.658658 26695 catalog_manager.cc:1259] Loaded cluster ID: 4c0afa372dac4077b1e53622c7454eba
I20260806 09:22:09.658831 26695 catalog_manager.cc:1552] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: loading cluster ID for follower catalog manager: success
I20260806 09:22:09.663710 26695 catalog_manager.cc:1574] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: acquiring CA information for follower catalog manager: success
I20260806 09:22:09.667121 26695 catalog_manager.cc:1602] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260806 09:22:09.973348 26615 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:09.973743 26615 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:09.973927 26615 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:09.981655 26615 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:09.981827 26615 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:09.982013 26615 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:22:09.993178 26615 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab
--principal=oryx/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:45867
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=41761
--tserver_master_addrs=127.16.86.190:43733,127.16.86.189:46587,127.16.86.188:33963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:09.994690 26615 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:09.996325 26615 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 09:22:10.003612 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.996898 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51094 (local address 127.16.86.188:33963)
0806 09:22:09.997803 (+ 905us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:09.997808 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:09.997825 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:22:09.997904 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:09.997908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:09.997994 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:09.998176 (+ 182us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:09.998662 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.999780 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.001412 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.002312 (+ 900us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.002480 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.002752 (+ 272us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:10.002989 (+ 237us) server_negotiation.cc:299] Negotiation successful
0806 09:22:10.003206 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":532,"threads_started":1}
I20260806 09:22:10.003615 26704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:09.996747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:33963 (local address 127.0.0.1:51094)
0806 09:22:09.997382 (+ 635us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:09.997413 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:09.997634 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:09.998197 (+ 563us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:09.998202 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:09.998231 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:09.998494 (+ 263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:09.998505 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:09.999954 (+ 1449us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:09.999959 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.001235 (+ 1276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.001248 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.002521 (+ 1273us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.002527 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.002747 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.002797 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 09:22:10.002944 (+ 147us) client_negotiation.cc:240] Negotiation successful
0806 09:22:10.003172 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":361,"threads_started":1}
I20260806 09:22:10.005559 26636 raft_consensus.cc:1203] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20260806 09:22:10.006721 26681 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b2fc5f140794fee87c356b29cd9fb6a. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } } }
I20260806 09:22:10.007112 26681 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: This master's current role is: LEARNER
W20260806 09:22:10.013135 26709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:10.013999 26712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:10.014891 26716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:10.014880 26615 server_base.cc:1034] running on GCE node
I20260806 09:22:10.015935 26615 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:10.017112 26615 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:10.018373 26615 hybrid_clock.cc:648] HybridClock initialized: now 1786008130018310 us; error 63 us; skew 500 ppm
I20260806 09:22:10.018777 26615 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:10 dist-test-slave-h3ft krb5kdc[25392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008130, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:10.027266 26615 init.cc:377] Logged in from keytab as oryx/127.16.86.129@KRBTEST.COM (short username oryx)
I20260806 09:22:10.029971 26615 webserver.cc:460] Webserver started at http://127.16.86.129:41761/ using document root <none> and password file <none>
I20260806 09:22:10.031070 26615 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:10.031270 26615 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:10.037907 26615 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:22:10.041738 26726 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:10.043870 26615 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:22:10.044226 26615 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "d3d9877aba6a445280ab89df34417458"
format_stamp: "Formatted at 2026-08-06 09:22:06 on dist-test-slave-h3ft"
server_key: "715b4dc5ad435655b47d19509ea38ccf"
server_key_iv: "5baa3e35ef30225a988760d9673a5eda"
server_key_version: "encryptionkey@0"
I20260806 09:22:10.045529 26615 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:10.081473 26615 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:10.086796 26615 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:10.087177 26615 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:10.088382 26615 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:10.090152 26615 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:10.090286 26615 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:10.090428 26615 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:10.090544 26615 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:10.132599 26615 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:45867
I20260806 09:22:10.132704 26884 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:45867 every 8 connection(s)
I20260806 09:22:10.134171 26615 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:22:10.138979 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 26615
I20260806 09:22:10.139546 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 25745
Aug 06 09:22:10 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.86.130@KRBTEST.COM
Aug 06 09:22:10 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008130, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
Aug 06 09:22:10 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008130, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.188@KRBTEST.COM
Aug 06 09:22:10 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008130, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.189@KRBTEST.COM
I20260806 09:22:10.185858 26895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.141668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51448 (local address 127.16.86.130:33775)
0806 09:22:10.156298 (+ 14630us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:10.156315 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:10.157382 (+ 1067us) server_negotiation.cc:407] Connection header received
0806 09:22:10.157495 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:10.157504 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:10.157822 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:10.158037 (+ 215us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:10.160754 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.162509 (+ 1755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.165538 (+ 3029us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.167206 (+ 1668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.167351 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.176347 (+ 8996us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:10.176369 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:10.176388 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:10.176444 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:10.181014 (+ 4570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.182117 (+ 1103us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.182129 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.182134 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.182303 (+ 169us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.182796 (+ 493us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.182802 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.182807 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.183365 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:10.183699 (+ 334us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:10.185094 (+ 1395us) server_negotiation.cc:299] Negotiation successful
0806 09:22:10.185388 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14365,"thread_start_us":12867,"threads_started":1}
I20260806 09:22:10.187508 25916 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:10.188035 26706 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.138229 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:45631 (local address 127.16.86.188:33963)
0806 09:22:10.138448 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:10.138454 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:10.157687 (+ 19233us) server_negotiation.cc:407] Connection header received
0806 09:22:10.160821 (+ 3134us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:10.160831 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:10.160949 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:10.161042 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:10.169160 (+ 8118us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.170457 (+ 1297us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.173215 (+ 2758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.173937 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.174044 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.179240 (+ 5196us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:10.179250 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:10.179255 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:10.179306 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:10.183354 (+ 4048us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.184688 (+ 1334us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.184694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.184698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.184873 (+ 175us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.185516 (+ 643us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.185522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.185525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.186262 (+ 737us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:10.186433 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:10.187329 (+ 896us) server_negotiation.cc:299] Negotiation successful
0806 09:22:10.187653 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 09:22:10.188087 26889 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.138195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:55781 (local address 127.16.86.190:43733)
0806 09:22:10.154642 (+ 16447us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:10.154652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:10.160913 (+ 6261us) server_negotiation.cc:407] Connection header received
0806 09:22:10.161012 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:10.161019 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:10.161133 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:10.161251 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:10.165181 (+ 3930us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.166773 (+ 1592us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.169768 (+ 2995us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.170431 (+ 663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.170538 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.178627 (+ 8089us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:10.178636 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:10.178642 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:10.178698 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:10.183406 (+ 4708us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.184688 (+ 1282us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.184694 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.184698 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.184889 (+ 191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.185516 (+ 627us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.185522 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.185525 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.185917 (+ 392us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:10.186080 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:10.187313 (+ 1233us) server_negotiation.cc:299] Negotiation successful
0806 09:22:10.187597 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16200,"thread_start_us":15941,"threads_started":1}
I20260806 09:22:10.191460 26892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.138745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.129:55781)
0806 09:22:10.156941 (+ 18196us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:10.157023 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:10.160384 (+ 3361us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:10.161699 (+ 1315us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:10.161713 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:10.164509 (+ 2796us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:10.164892 (+ 383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.164908 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.167078 (+ 2170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.167087 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.169590 (+ 2503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.169604 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.170979 (+ 1375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.170990 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.171386 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.173071 (+ 1685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:10.173110 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:10.178218 (+ 5108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:10.183568 (+ 5350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:10.183572 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:10.183597 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:10.184529 (+ 932us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:10.185039 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:10.185048 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:10.185052 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:10.185380 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:10.186177 (+ 797us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:10.186186 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:10.186777 (+ 591us) client_negotiation.cc:769] Sending connection context
0806 09:22:10.188433 (+ 1656us) client_negotiation.cc:240] Negotiation successful
0806 09:22:10.189716 (+ 1283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16522,"mutex_wait_us":498,"spinlock_wait_cycles":36864,"thread_start_us":24177,"threads_started":1}
I20260806 09:22:10.192170 26891 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.139212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:33963 (local address 127.16.86.129:45631)
0806 09:22:10.157510 (+ 18298us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:10.157540 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:10.160397 (+ 2857us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:10.161706 (+ 1309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:10.161715 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:10.162982 (+ 1267us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:10.164171 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.164209 (+ 38us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.170804 (+ 6595us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.170811 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.172999 (+ 2188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.173021 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.174452 (+ 1431us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.174459 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.174698 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.175546 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:10.175572 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:10.178978 (+ 3406us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:10.183560 (+ 4582us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:10.183571 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:10.183597 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:10.184529 (+ 932us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:10.185039 (+ 510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:10.185044 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:10.185052 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:10.185373 (+ 321us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:10.186508 (+ 1135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:10.186512 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:10.186777 (+ 265us) client_negotiation.cc:769] Sending connection context
0806 09:22:10.188812 (+ 2035us) client_negotiation.cc:240] Negotiation successful
0806 09:22:10.189363 (+ 551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":18092,"mutex_wait_us":111,"spinlock_wait_cycles":49536,"thread_start_us":15842,"threads_started":1}
I20260806 09:22:10.192961 26896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.156898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46587 (local address 127.16.86.129:38157)
0806 09:22:10.169027 (+ 12129us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:10.169049 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:10.169202 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:10.169796 (+ 594us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:10.169800 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:10.170402 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:10.170775 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.170789 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.172227 (+ 1438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.172234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.173765 (+ 1531us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.173780 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.175283 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.175295 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.176368 (+ 1073us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.177178 (+ 810us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:10.177208 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:10.180382 (+ 3174us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:10.184757 (+ 4375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:10.184763 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:10.184766 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:10.185419 (+ 653us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:10.185887 (+ 468us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:10.185893 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:10.185897 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:10.186078 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:10.186807 (+ 729us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:10.186812 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:10.186990 (+ 178us) client_negotiation.cc:769] Sending connection context
0806 09:22:10.192021 (+ 5031us) client_negotiation.cc:240] Negotiation successful
0806 09:22:10.192417 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11977,"mutex_wait_us":77,"thread_start_us":11788,"threads_started":1}
I20260806 09:22:10.194342 26885 heartbeater.cc:347] Connected to a master server at 127.16.86.188:33963
I20260806 09:22:10.194401 26886 heartbeater.cc:347] Connected to a master server at 127.16.86.190:43733
I20260806 09:22:10.194731 26898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.157055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:38157 (local address 127.16.86.189:46587)
0806 09:22:10.168400 (+ 11345us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:10.168406 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:10.169311 (+ 905us) server_negotiation.cc:407] Connection header received
0806 09:22:10.169381 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:10.169387 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:10.169467 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:10.169570 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:10.170950 (+ 1380us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.172059 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.173959 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.174850 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.174997 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.180671 (+ 5674us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:10.180679 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:10.180684 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:10.180732 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:10.184581 (+ 3849us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.185568 (+ 987us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.185588 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.185593 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.185738 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.186223 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.186228 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.186231 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.186568 (+ 337us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:10.186735 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:10.193453 (+ 6718us) server_negotiation.cc:299] Negotiation successful
0806 09:22:10.194348 (+ 895us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11201,"thread_start_us":11341,"threads_started":1}
I20260806 09:22:10.194761 26885 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:10.194766 26886 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:10.195875 26887 heartbeater.cc:347] Connected to a master server at 127.16.86.189:46587
I20260806 09:22:10.195922 26886 heartbeater.cc:511] Master 127.16.86.190:43733 requested a full tablet report, sending...
I20260806 09:22:10.196055 26887 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:10.196722 26887 heartbeater.cc:511] Master 127.16.86.189:46587 requested a full tablet report, sending...
I20260806 09:22:10.197292 26885 heartbeater.cc:511] Master 127.16.86.188:33963 requested a full tablet report, sending...
I20260806 09:22:10.198252 25460 ts_manager.cc:194] Re-registered known tserver with Master: d3d9877aba6a445280ab89df34417458 (127.16.86.129:45867)
I20260806 09:22:10.199333 25460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:55781
I20260806 09:22:10.199461 26620 ts_manager.cc:194] Registered new tserver with Master: d3d9877aba6a445280ab89df34417458 (127.16.86.129:45867)
I20260806 09:22:10.200244 26327 ts_manager.cc:194] Registered new tserver with Master: d3d9877aba6a445280ab89df34417458 (127.16.86.129:45867)
I20260806 09:22:10.284235 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 25745
I20260806 09:22:10.313380 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:33775
--local_ip_for_outbound_sockets=127.16.86.130
--tserver_master_addrs=127.16.86.190:43733
--webserver_port=41065
--webserver_interface=127.16.86.130
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab
--principal=oryx/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.86.190:43733,127.16.86.189:46587
--tserver_master_addrs=127.16.86.190:43733,127.16.86.189:46587,127.16.86.188:33963 with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:22:10.548130 26557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: NON_VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: true } } }
I20260806 09:22:10.548662 26557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:10.551286 26902 raft_consensus.cc:1050] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a: attempting to promote NON_VOTER dee9589594ac4444bff7783dc8b59f3b to VOTER
I20260806 09:22:10.552001 26902 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } }
I20260806 09:22:10.554729 26342 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 FOLLOWER]: Refusing update from remote peer 4b2fc5f140794fee87c356b29cd9fb6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 09:22:10.555162 26636 raft_consensus.cc:1261] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 1 LEARNER]: Refusing update from remote peer 4b2fc5f140794fee87c356b29cd9fb6a: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20260806 09:22:10.555580 26902 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 09:22:10.555979 26903 consensus_queue.cc:1035] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [LEADER]: Connected to new peer: Peer: permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20260806 09:22:10.561098 26902 raft_consensus.cc:2932] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dee9589594ac4444bff7783dc8b59f3b (127.16.86.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } } }
I20260806 09:22:10.561755 26342 raft_consensus.cc:2932] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dee9589594ac4444bff7783dc8b59f3b (127.16.86.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } } }
I20260806 09:22:10.565892 26557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } } }
I20260806 09:22:10.565969 26903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } } }
I20260806 09:22:10.566201 26557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:10.566231 26903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:10.567893 26903 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } } }
I20260806 09:22:10.568145 26903 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:10.568746 26909 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } } }
I20260806 09:22:10.569234 26909 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:22:10.568733 26636 raft_consensus.cc:2932] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, dee9589594ac4444bff7783dc8b59f3b (127.16.86.188) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } } }
I20260806 09:22:10.572944 26910 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } } }
I20260806 09:22:10.573278 26910 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: This master's current role is: FOLLOWER
W20260806 09:22:10.685040 26900 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:10.685374 26900 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:10.685531 26900 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:10.694375 26900 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:10.694509 26900 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:10.694703 26900 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:22:10.705631 26900 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab
--principal=oryx/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:33775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=41065
--tserver_master_addrs=127.16.86.190:43733,127.16.86.189:46587,127.16.86.188:33963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:10.707208 26900 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:10.708657 26900 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:10.725984 26919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:10.726042 26918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:10.726084 26922 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:10.727804 26900 server_base.cc:1034] running on GCE node
I20260806 09:22:10.728597 26900 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:10.729785 26900 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:10.731110 26900 hybrid_clock.cc:648] HybridClock initialized: now 1786008130731032 us; error 61 us; skew 500 ppm
I20260806 09:22:10.731487 26900 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:10 dist-test-slave-h3ft krb5kdc[25392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008130, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:10.735697 26900 init.cc:377] Logged in from keytab as oryx/127.16.86.130@KRBTEST.COM (short username oryx)
I20260806 09:22:10.737888 26900 webserver.cc:460] Webserver started at http://127.16.86.130:41065/ using document root <none> and password file <none>
I20260806 09:22:10.738638 26900 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:10.738780 26900 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:10.745272 26900 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260806 09:22:10.749118 26933 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:10.751592 26900 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:22:10.751845 26900 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "5cb7873ac0eb467ebce54384aee30caa"
format_stamp: "Formatted at 2026-08-06 09:22:06 on dist-test-slave-h3ft"
server_key: "1a9b48fe0fde2ba84727bac8c65c20c7"
server_key_iv: "9e71f5a581846967c96ee842f2ee9ca4"
server_key_version: "encryptionkey@0"
I20260806 09:22:10.753137 26900 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:10.774575 26900 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:10.779814 26900 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:10.780133 26900 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:10.781317 26900 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:10.782933 26900 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:10.783084 26900 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 09:22:10.783236 26900 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:10.783334 26900 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:10.825568 26900 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:33775
I20260806 09:22:10.825713 27082 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:33775 every 8 connection(s)
I20260806 09:22:10.827325 26900 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:22:10.837843 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 26900
I20260806 09:22:10.838224 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 25931
Aug 06 09:22:10 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008130, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:10.865811 27090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.831399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:56437 (local address 127.16.86.190:43733)
0806 09:22:10.832144 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:10.832155 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:10.833985 (+ 1830us) server_negotiation.cc:407] Connection header received
0806 09:22:10.837130 (+ 3145us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:10.837136 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:10.837266 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:10.837398 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:10.840947 (+ 3549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.842299 (+ 1352us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.846904 (+ 4605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.847869 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.848029 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.855575 (+ 7546us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:10.855585 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:10.855593 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:10.855660 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:10.860530 (+ 4870us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.862031 (+ 1501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.862037 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.862041 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.862322 (+ 281us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.863089 (+ 767us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.863095 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.863099 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.863420 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:10.863610 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:10.864971 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0806 09:22:10.865334 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":886,"threads_started":1}
Aug 06 09:22:10 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008130, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.189@KRBTEST.COM
Aug 06 09:22:10 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008130, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.188@KRBTEST.COM
I20260806 09:22:10.867690 27092 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.832263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.130:56437)
0806 09:22:10.833890 (+ 1627us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:10.833912 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:10.836750 (+ 2838us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:10.838467 (+ 1717us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:10.838475 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:10.839206 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:10.840711 (+ 1505us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.840759 (+ 48us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.842924 (+ 2165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.842934 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.846569 (+ 3635us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.846631 (+ 62us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.848349 (+ 1718us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.848414 (+ 65us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.849056 (+ 642us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.850408 (+ 1352us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:10.850468 (+ 60us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:10.855212 (+ 4744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:10.860737 (+ 5525us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:10.860753 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:10.860783 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:10.861863 (+ 1080us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:10.862465 (+ 602us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:10.862473 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:10.862478 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:10.862907 (+ 429us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:10.863587 (+ 680us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:10.863596 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:10.864375 (+ 779us) client_negotiation.cc:769] Sending connection context
0806 09:22:10.864884 (+ 509us) client_negotiation.cc:240] Negotiation successful
0806 09:22:10.865563 (+ 679us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1418,"mutex_wait_us":297,"spinlock_wait_cycles":128512,"thread_start_us":972,"threads_started":1}
I20260806 09:22:10.872917 27086 heartbeater.cc:347] Connected to a master server at 127.16.86.190:43733
I20260806 09:22:10.873365 27086 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:10.874639 27086 heartbeater.cc:511] Master 127.16.86.190:43733 requested a full tablet report, sending...
I20260806 09:22:10.877439 25460 ts_manager.cc:194] Re-registered known tserver with Master: 5cb7873ac0eb467ebce54384aee30caa (127.16.86.130:33775)
I20260806 09:22:10.878831 25460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:56437
I20260806 09:22:10.882155 27091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.832263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:33963 (local address 127.16.86.130:54129)
0806 09:22:10.833610 (+ 1347us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:10.833695 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:10.836750 (+ 3055us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:10.838388 (+ 1638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:10.838394 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:10.839329 (+ 935us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:10.840712 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.840753 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.843849 (+ 3096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.843856 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.846245 (+ 2389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.846270 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.860191 (+ 13921us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.860201 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.860494 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.861699 (+ 1205us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:10.861731 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:10.873599 (+ 11868us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:10.878632 (+ 5033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:10.878639 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:10.878644 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:10.879389 (+ 745us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:10.879946 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:10.879951 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:10.879957 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:10.880169 (+ 212us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:10.880986 (+ 817us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:10.880991 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:10.881148 (+ 157us) client_negotiation.cc:769] Sending connection context
0806 09:22:10.881312 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 09:22:10.881668 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":935,"mutex_wait_us":4,"spinlock_wait_cycles":6656,"thread_start_us":545,"threads_started":1}
I20260806 09:22:10.883343 27089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.831439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:54129 (local address 127.16.86.188:33963)
0806 09:22:10.832609 (+ 1170us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:10.832616 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:10.833857 (+ 1241us) server_negotiation.cc:407] Connection header received
0806 09:22:10.837746 (+ 3889us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:10.837753 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:10.837853 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:10.837951 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:10.841425 (+ 3474us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.842827 (+ 1402us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.858411 (+ 15584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.859571 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.859755 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.874015 (+ 14260us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:10.874025 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:10.874031 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:10.874086 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:10.878409 (+ 4323us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.879581 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.879587 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.879591 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.879775 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.880325 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.880330 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.880333 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.880742 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:10.880914 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:10.882556 (+ 1642us) server_negotiation.cc:299] Negotiation successful
0806 09:22:10.882845 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":997,"thread_start_us":866,"threads_started":1}
I20260806 09:22:10.884413 27084 heartbeater.cc:347] Connected to a master server at 127.16.86.188:33963
I20260806 09:22:10.884590 27084 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:10.885170 27084 heartbeater.cc:511] Master 127.16.86.188:33963 requested a full tablet report, sending...
I20260806 09:22:10.886451 26620 ts_manager.cc:194] Registered new tserver with Master: 5cb7873ac0eb467ebce54384aee30caa (127.16.86.130:33775)
I20260806 09:22:10.886610 27093 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.834657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46587 (local address 127.16.86.130:53427)
0806 09:22:10.835476 (+ 819us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:10.835502 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:10.836770 (+ 1268us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:10.837800 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:10.837810 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:10.839090 (+ 1280us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:10.841154 (+ 2064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.841175 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.843140 (+ 1965us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.843149 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.855428 (+ 12279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:10.855461 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.857498 (+ 2037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.857510 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:10.857823 (+ 313us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.861078 (+ 3255us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:10.861115 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:10.867651 (+ 6536us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:10.873876 (+ 6225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:10.873884 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:10.873891 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:10.874879 (+ 988us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:10.883210 (+ 8331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:10.883217 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:10.883220 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:10.883670 (+ 450us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:10.884805 (+ 1135us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:10.884813 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:10.885085 (+ 272us) client_negotiation.cc:769] Sending connection context
0806 09:22:10.885261 (+ 176us) client_negotiation.cc:240] Negotiation successful
0806 09:22:10.885887 (+ 626us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"mutex_wait_us":5916,"thread_start_us":452,"threads_started":1}
I20260806 09:22:10.887688 27094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.834724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:53427 (local address 127.16.86.189:46587)
0806 09:22:10.835952 (+ 1228us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:10.835958 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:10.835978 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:22:10.837149 (+ 1171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:10.837155 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:10.837283 (+ 128us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:10.837438 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:10.841396 (+ 3958us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.842648 (+ 1252us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.855760 (+ 13112us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.856857 (+ 1097us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.857056 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.868439 (+ 11383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:10.868448 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:10.868456 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:10.868539 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:10.872270 (+ 3731us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.876195 (+ 3925us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.876206 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.876213 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.876446 (+ 233us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.883887 (+ 7441us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.883898 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.883904 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.884474 (+ 570us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:10.884695 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:10.885533 (+ 838us) server_negotiation.cc:299] Negotiation successful
0806 09:22:10.887370 (+ 1837us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1109,"thread_start_us":990,"threads_started":1}
I20260806 09:22:10.890658 27087 heartbeater.cc:347] Connected to a master server at 127.16.86.189:46587
I20260806 09:22:10.890832 27087 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:10.891292 27087 heartbeater.cc:511] Master 127.16.86.189:46587 requested a full tablet report, sending...
I20260806 09:22:10.893016 26327 ts_manager.cc:194] Registered new tserver with Master: 5cb7873ac0eb467ebce54384aee30caa (127.16.86.130:33775)
Aug 06 09:22:10 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008124, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.86.131@KRBTEST.COM
I20260806 09:22:10.909116 27097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:10.839912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.16.86.131:36101)
0806 09:22:10.875878 (+ 35966us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:10.875889 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:10.876626 (+ 737us) server_negotiation.cc:407] Connection header received
0806 09:22:10.876727 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:10.876740 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:10.877154 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:10.877415 (+ 261us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:10.880492 (+ 3077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.882688 (+ 2196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.888900 (+ 6212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:10.890527 (+ 1627us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:10.890750 (+ 223us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:10.898827 (+ 8077us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:10.898858 (+ 31us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:10.898894 (+ 36us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:10.898973 (+ 79us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:10.904282 (+ 5309us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.905431 (+ 1149us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.905439 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.905442 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.905637 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:10.906394 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:10.906403 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:10.906408 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:10.907351 (+ 943us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:10.907907 (+ 556us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:10.908235 (+ 328us) server_negotiation.cc:299] Negotiation successful
0806 09:22:10.908684 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":26664,"thread_start_us":26379,"threads_started":1}
I20260806 09:22:10.910897 26102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:11.004173 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 25931
I20260806 09:22:11.031376 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:36101
--local_ip_for_outbound_sockets=127.16.86.131
--tserver_master_addrs=127.16.86.190:43733
--webserver_port=35881
--webserver_interface=127.16.86.131
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab
--principal=oryx/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33519
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.16.86.190:43733,127.16.86.189:46587
--tserver_master_addrs=127.16.86.190:43733,127.16.86.189:46587,127.16.86.188:33963 with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:22:11.202747 26886 heartbeater.cc:503] Master 127.16.86.190:43733 was elected leader, sending a full tablet report...
W20260806 09:22:11.464773 27099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:11.465216 27099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:11.465438 27099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:11.473953 27099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:11.474112 27099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:11.474350 27099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:22:11.485785 27099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33519
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab
--principal=oryx/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:36101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=35881
--tserver_master_addrs=127.16.86.190:43733,127.16.86.189:46587,127.16.86.188:33963
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:11.487884 27099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:11.489678 27099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:11.506018 27109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:11.506094 27108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:11.507475 27113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:11.507475 27099 server_base.cc:1034] running on GCE node
I20260806 09:22:11.508875 27099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:11.510257 27099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:11.511551 27099 hybrid_clock.cc:648] HybridClock initialized: now 1786008131511481 us; error 34 us; skew 500 ppm
I20260806 09:22:11.512018 27099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:11 dist-test-slave-h3ft krb5kdc[25392](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008131, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:11.516314 27099 init.cc:377] Logged in from keytab as oryx/127.16.86.131@KRBTEST.COM (short username oryx)
I20260806 09:22:11.518376 27099 webserver.cc:460] Webserver started at http://127.16.86.131:35881/ using document root <none> and password file <none>
I20260806 09:22:11.519347 27099 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:11.519541 27099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:11.526940 27099 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.003s sys 0.003s
I20260806 09:22:11.530755 27120 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:11.532691 27099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:22:11.532941 27099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "d8f51643bf134d19843fc65d7fd39502"
format_stamp: "Formatted at 2026-08-06 09:22:07 on dist-test-slave-h3ft"
server_key: "303cfd88d8045929a201c3bda7c20a0a"
server_key_iv: "946dc5743e3d588d184dae32cc717939"
server_key_version: "encryptionkey@0"
I20260806 09:22:11.534184 27099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:11.557564 27099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:11.563069 27099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:11.563377 27099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:11.564523 27099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:11.566355 27099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:11.566490 27099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:11.566636 27099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:11.566731 27099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:11.610652 27099 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:36101
I20260806 09:22:11.610695 27283 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:36101 every 8 connection(s)
I20260806 09:22:11.612660 27099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:22:11.618250 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 27099
Aug 06 09:22:11 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008131, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.189@KRBTEST.COM
Aug 06 09:22:11 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008131, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
Aug 06 09:22:11 dist-test-slave-h3ft krb5kdc[25392](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008131, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.188@KRBTEST.COM
I20260806 09:22:11.674324 27306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.639519 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46587 (local address 127.16.86.131:59053)
0806 09:22:11.643319 (+ 3800us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:11.643342 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:11.643496 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:11.644085 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:11.644090 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:11.644760 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:11.646632 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.646672 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.650638 (+ 3966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.650642 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.654151 (+ 3509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.654168 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.656430 (+ 2262us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.656442 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.656680 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.657392 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:11.657417 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:11.660614 (+ 3197us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:11.665849 (+ 5235us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:11.665863 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:11.665889 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:11.666532 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:11.667948 (+ 1416us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:11.667955 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:11.667960 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:11.668290 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:11.670622 (+ 2332us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:11.670636 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:11.671329 (+ 693us) client_negotiation.cc:769] Sending connection context
0806 09:22:11.671739 (+ 410us) client_negotiation.cc:240] Negotiation successful
0806 09:22:11.672496 (+ 757us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3641,"mutex_wait_us":2,"thread_start_us":4058,"threads_started":1}
I20260806 09:22:11.675668 27303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.637062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:59053 (local address 127.16.86.189:46587)
0806 09:22:11.640156 (+ 3094us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:11.640166 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:11.643444 (+ 3278us) server_negotiation.cc:407] Connection header received
0806 09:22:11.643708 (+ 264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:11.643717 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:11.643849 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:11.644027 (+ 178us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:11.646963 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.648278 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.654361 (+ 6083us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.656057 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.656363 (+ 306us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.660974 (+ 4611us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:11.660985 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:11.660990 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:11.661045 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:11.665682 (+ 4637us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:11.666677 (+ 995us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:11.666686 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:11.666693 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:11.666887 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:11.669614 (+ 2727us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:11.669621 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:11.669626 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:11.670325 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:11.670503 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:11.675055 (+ 4552us) server_negotiation.cc:299] Negotiation successful
0806 09:22:11.675381 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2874,"thread_start_us":2168,"threads_started":1}
I20260806 09:22:11.677445 27292 heartbeater.cc:347] Connected to a master server at 127.16.86.189:46587
I20260806 09:22:11.677901 27292 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:11.679320 27292 heartbeater.cc:511] Master 127.16.86.189:46587 requested a full tablet report, sending...
I20260806 09:22:11.681773 26327 ts_manager.cc:194] Registered new tserver with Master: d8f51643bf134d19843fc65d7fd39502 (127.16.86.131:36101)
I20260806 09:22:11.682353 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.617563 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:36301 (local address 127.16.86.190:43733)
0806 09:22:11.645216 (+ 27653us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:11.645223 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:11.645246 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:22:11.645320 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:11.645325 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:11.645409 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:11.645538 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:11.647169 (+ 1631us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.648483 (+ 1314us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.651495 (+ 3012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.652228 (+ 733us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.652355 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.666865 (+ 14510us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:11.666875 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:11.666880 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:11.666940 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:11.673915 (+ 6975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:11.677548 (+ 3633us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:11.677555 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:11.677558 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:11.677738 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:11.679160 (+ 1422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:11.679166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:11.679169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:11.679666 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:11.679814 (+ 148us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:11.681561 (+ 1747us) server_negotiation.cc:299] Negotiation successful
0806 09:22:11.681894 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":27492,"thread_start_us":27346,"threads_started":1}
I20260806 09:22:11.683786 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.618589 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.131:36301)
0806 09:22:11.635877 (+ 17288us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:11.635948 (+ 71us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:11.644205 (+ 8257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:11.645735 (+ 1530us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:11.645740 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:11.646312 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:11.646877 (+ 565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.646893 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.648666 (+ 1773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.648671 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.650811 (+ 2140us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.650829 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.653325 (+ 2496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.653333 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.653555 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.655080 (+ 1525us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:11.655143 (+ 63us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:11.664756 (+ 9613us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:11.676749 (+ 11993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:11.676754 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:11.676757 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:11.677332 (+ 575us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:11.678348 (+ 1016us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:11.678413 (+ 65us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:11.678418 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:11.678635 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:11.679917 (+ 1282us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:11.679921 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:11.680096 (+ 175us) client_negotiation.cc:769] Sending connection context
0806 09:22:11.683101 (+ 3005us) client_negotiation.cc:240] Negotiation successful
0806 09:22:11.683464 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16825,"mutex_wait_us":420,"thread_start_us":20891,"threads_started":1}
I20260806 09:22:11.684984 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.617609 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:45945 (local address 127.16.86.188:33963)
0806 09:22:11.630724 (+ 13115us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:11.630734 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:11.639443 (+ 8709us) server_negotiation.cc:407] Connection header received
0806 09:22:11.641085 (+ 1642us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:11.641097 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:11.641245 (+ 148us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:11.641371 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:11.646510 (+ 5139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.647806 (+ 1296us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.651050 (+ 3244us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.651759 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.651873 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.667374 (+ 15501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:11.667383 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:11.667388 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:11.667447 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:11.672034 (+ 4587us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:11.677933 (+ 5899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:11.677940 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:11.677945 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:11.678182 (+ 237us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:11.680329 (+ 2147us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:11.680337 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:11.680342 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:11.680773 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:11.680918 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:11.684200 (+ 3282us) server_negotiation.cc:299] Negotiation successful
0806 09:22:11.684563 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12893,"thread_start_us":12737,"threads_started":1}
I20260806 09:22:11.685432 27290 heartbeater.cc:347] Connected to a master server at 127.16.86.190:43733
I20260806 09:22:11.685590 27290 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:11.686182 27290 heartbeater.cc:511] Master 127.16.86.190:43733 requested a full tablet report, sending...
I20260806 09:22:11.686266 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.619329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:33963 (local address 127.16.86.131:45945)
0806 09:22:11.639283 (+ 19954us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:11.639307 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:11.640536 (+ 1229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:11.642267 (+ 1731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:11.642289 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:11.644891 (+ 2602us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:11.646208 (+ 1317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.646249 (+ 41us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.648043 (+ 1794us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.648049 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.650622 (+ 2573us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.650636 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.652326 (+ 1690us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.652335 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.652721 (+ 386us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.657092 (+ 4371us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:11.657185 (+ 93us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:11.665763 (+ 8578us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:11.676005 (+ 10242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:11.676011 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:11.676016 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:11.676606 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:11.678779 (+ 2173us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:11.678783 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:11.678786 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:11.678965 (+ 179us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:11.681034 (+ 2069us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:11.681039 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:11.681205 (+ 166us) client_negotiation.cc:769] Sending connection context
0806 09:22:11.685327 (+ 4122us) client_negotiation.cc:240] Negotiation successful
0806 09:22:11.685706 (+ 379us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19763,"mutex_wait_us":64,"thread_start_us":15965,"threads_started":1}
I20260806 09:22:11.687326 27289 heartbeater.cc:347] Connected to a master server at 127.16.86.188:33963
I20260806 09:22:11.687497 27289 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:11.687963 27289 heartbeater.cc:511] Master 127.16.86.188:33963 requested a full tablet report, sending...
I20260806 09:22:11.687963 25460 ts_manager.cc:194] Re-registered known tserver with Master: d8f51643bf134d19843fc65d7fd39502 (127.16.86.131:36101)
I20260806 09:22:11.689045 25460 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:36301
I20260806 09:22:11.694658 26620 ts_manager.cc:194] Registered new tserver with Master: d8f51643bf134d19843fc65d7fd39502 (127.16.86.131:36101)
I20260806 09:22:11.710345 27294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.648689 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.16.86.190:43733)
0806 09:22:11.682659 (+ 33970us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:11.682665 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:11.682683 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:22:11.682734 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:11.682741 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:11.682822 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:11.682927 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:11.692444 (+ 9517us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.693641 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.696005 (+ 2364us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.696698 (+ 693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.696837 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.702515 (+ 5678us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:11.702527 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:11.702533 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:11.702591 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:11.706509 (+ 3918us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:11.707741 (+ 1232us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:11.707747 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:11.707752 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:11.707895 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:11.708331 (+ 436us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:11.708336 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:11.708339 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:11.708802 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:11.709093 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:11.709687 (+ 594us) server_negotiation.cc:299] Negotiation successful
0806 09:22:11.709947 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":33838,"thread_start_us":44694,"threads_started":1}
I20260806 09:22:11.712046 27311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.654366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53146 (local address 127.16.86.189:46587)
0806 09:22:11.656972 (+ 2606us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:11.656981 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:11.684565 (+ 27584us) server_negotiation.cc:407] Connection header received
0806 09:22:11.684765 (+ 200us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:11.684772 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:11.684856 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:11.684991 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:11.689557 (+ 4566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.690690 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.695081 (+ 4391us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.695685 (+ 604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.695844 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.701294 (+ 5450us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:11.701302 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:11.701306 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:11.701359 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:11.705585 (+ 4226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:11.708184 (+ 2599us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:11.708191 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:11.708195 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:11.708453 (+ 258us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:11.708922 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:11.708927 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:11.708930 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:11.709360 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:11.709669 (+ 309us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:11.711278 (+ 1609us) server_negotiation.cc:299] Negotiation successful
0806 09:22:11.711570 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2423,"thread_start_us":2215,"threads_started":1}
I20260806 09:22:11.713785 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.672375 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51108 (local address 127.16.86.188:33963)
0806 09:22:11.685279 (+ 12904us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:11.685285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:11.693146 (+ 7861us) server_negotiation.cc:407] Connection header received
0806 09:22:11.693887 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:11.693894 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:11.694044 (+ 150us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:11.694127 (+ 83us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:11.697732 (+ 3605us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.698868 (+ 1136us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.701245 (+ 2377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.701969 (+ 724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.702162 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.705260 (+ 3098us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:11.705266 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:11.705269 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:11.705313 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:11.708347 (+ 3034us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:11.709219 (+ 872us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:11.709224 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:11.709228 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:11.709377 (+ 149us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:11.709890 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:11.709896 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:11.709899 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:11.710332 (+ 433us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:11.710606 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:11.711899 (+ 1293us) server_negotiation.cc:299] Negotiation successful
0806 09:22:11.713450 (+ 1551us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12779,"thread_start_us":13702,"threads_started":1}
I20260806 09:22:11.715646 25460 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53826:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:22:11.723867 25460 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:22:11.769872 27332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.748415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:45867 (local address 127.0.0.1:52380)
0806 09:22:11.760752 (+ 12337us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:11.760781 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:11.760984 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:11.761816 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:11.761821 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:11.761846 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:11.762387 (+ 541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.762399 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.763976 (+ 1577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.763982 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.765972 (+ 1990us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.765986 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.767877 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.767883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.768083 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.768132 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 09:22:11.768233 (+ 101us) client_negotiation.cc:240] Negotiation successful
0806 09:22:11.768380 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12195,"spinlock_wait_cycles":29056,"thread_start_us":14660,"threads_started":1}
I20260806 09:22:11.769943 27338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.749022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52380 (local address 127.16.86.129:45867)
0806 09:22:11.759670 (+ 10648us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:11.759682 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:11.761129 (+ 1447us) server_negotiation.cc:407] Connection header received
0806 09:22:11.761287 (+ 158us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:11.761298 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:11.761555 (+ 257us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:11.761797 (+ 242us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:11.762562 (+ 765us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.763781 (+ 1219us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.766170 (+ 2389us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.767353 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.767501 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.767747 (+ 246us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:11.769259 (+ 1512us) server_negotiation.cc:299] Negotiation successful
0806 09:22:11.769485 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10391,"thread_start_us":9839,"threads_started":1}
I20260806 09:22:11.774036 27335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.748904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:33775 (local address 127.0.0.1:51456)
0806 09:22:11.762746 (+ 13842us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:11.762780 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:11.763748 (+ 968us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:11.765460 (+ 1712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:11.765467 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:11.765510 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:11.766178 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.766196 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.768308 (+ 2112us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.768316 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.770916 (+ 2600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.770931 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.772972 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.772978 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.773152 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.773208 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 09:22:11.773336 (+ 128us) client_negotiation.cc:240] Negotiation successful
0806 09:22:11.773506 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13639,"mutex_wait_us":332,"thread_start_us":11485,"threads_started":1}
I20260806 09:22:11.774962 27339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.749317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51456 (local address 127.16.86.130:33775)
0806 09:22:11.759849 (+ 10532us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:11.759860 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:11.762986 (+ 3126us) server_negotiation.cc:407] Connection header received
0806 09:22:11.764052 (+ 1066us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:11.764067 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:11.764538 (+ 471us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:11.764856 (+ 318us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:11.766428 (+ 1572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.768106 (+ 1678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.771164 (+ 3058us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.772738 (+ 1574us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.772894 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.773159 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:11.774479 (+ 1320us) server_negotiation.cc:299] Negotiation successful
0806 09:22:11.774667 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10313,"thread_start_us":10529,"threads_started":1}
I20260806 09:22:11.776327 26792 tablet_service.cc:1467] Processing CreateTablet for tablet 1a249d912b7d45ecab79d81bfbe347aa (DEFAULT_TABLE table=test-table [id=e9120faf946a4ae29f1c90a9ad6fc78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:11.777724 26792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a249d912b7d45ecab79d81bfbe347aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:11.779294 26999 tablet_service.cc:1467] Processing CreateTablet for tablet 1a249d912b7d45ecab79d81bfbe347aa (DEFAULT_TABLE table=test-table [id=e9120faf946a4ae29f1c90a9ad6fc78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:11.780066 27334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.748315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38942 (local address 127.16.86.131:36101)
0806 09:22:11.767738 (+ 19423us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:11.767747 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:11.767789 (+ 42us) server_negotiation.cc:407] Connection header received
0806 09:22:11.767903 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:11.767911 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:11.768634 (+ 723us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:11.768863 (+ 229us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:11.770441 (+ 1578us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.771983 (+ 1542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.775228 (+ 3245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.778632 (+ 3404us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.778819 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.779161 (+ 342us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:11.779466 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 09:22:11.779666 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19203,"thread_start_us":18997,"threads_started":1}
I20260806 09:22:11.780753 27337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.747960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36101 (local address 127.0.0.1:38942)
0806 09:22:11.760750 (+ 12790us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:11.760775 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:11.760982 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:11.768840 (+ 7858us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:11.768845 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:11.768877 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:11.769506 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.769521 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.772145 (+ 2624us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.772150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.774501 (+ 2351us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.774513 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.778965 (+ 4452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.778972 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.779215 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.779266 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 09:22:11.779397 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 09:22:11.779568 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12603,"mutex_wait_us":44,"thread_start_us":14047,"threads_started":1}
I20260806 09:22:11.781170 26999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a249d912b7d45ecab79d81bfbe347aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:11.785809 27189 tablet_service.cc:1467] Processing CreateTablet for tablet 1a249d912b7d45ecab79d81bfbe347aa (DEFAULT_TABLE table=test-table [id=e9120faf946a4ae29f1c90a9ad6fc78d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:11.787738 27189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a249d912b7d45ecab79d81bfbe347aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:11.792824 27340 tablet_bootstrap.cc:492] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458: Bootstrap starting.
I20260806 09:22:11.796826 27340 tablet_bootstrap.cc:654] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:11.798008 27340 log.cc:824] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:11.800290 27343 tablet_bootstrap.cc:492] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa: Bootstrap starting.
I20260806 09:22:11.806093 27344 tablet_bootstrap.cc:492] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502: Bootstrap starting.
I20260806 09:22:11.806885 27343 tablet_bootstrap.cc:654] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:11.807749 27340 tablet_bootstrap.cc:492] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458: No bootstrap required, opened a new log
I20260806 09:22:11.808101 27340 ts_tablet_manager.cc:1397] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458: Time spent bootstrapping tablet: real 0.016s user 0.009s sys 0.004s
I20260806 09:22:11.808454 27343 log.cc:824] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:11.810253 27344 tablet_bootstrap.cc:654] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:11.811506 27344 log.cc:824] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:11.812433 27343 tablet_bootstrap.cc:492] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa: No bootstrap required, opened a new log
I20260806 09:22:11.812811 27343 ts_tablet_manager.cc:1397] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.000s
I20260806 09:22:11.814730 27344 tablet_bootstrap.cc:492] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502: No bootstrap required, opened a new log
I20260806 09:22:11.815132 27344 ts_tablet_manager.cc:1397] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502: Time spent bootstrapping tablet: real 0.009s user 0.009s sys 0.000s
I20260806 09:22:11.820439 27340 raft_consensus.cc:348] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:11.820976 27340 raft_consensus.cc:374] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:11.821090 27340 raft_consensus.cc:729] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d9877aba6a445280ab89df34417458, State: Initialized, Role: FOLLOWER
I20260806 09:22:11.821903 27340 consensus_queue.cc:260] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:11.824121 27340 ts_tablet_manager.cc:1428] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260806 09:22:11.825023 27343 raft_consensus.cc:348] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:11.826004 27343 raft_consensus.cc:374] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:11.826188 27343 raft_consensus.cc:729] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb7873ac0eb467ebce54384aee30caa, State: Initialized, Role: FOLLOWER
I20260806 09:22:11.826848 27343 consensus_queue.cc:260] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:11.825150 27344 raft_consensus.cc:348] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:11.830330 27344 raft_consensus.cc:374] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:11.830480 27344 raft_consensus.cc:729] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f51643bf134d19843fc65d7fd39502, State: Initialized, Role: FOLLOWER
I20260806 09:22:11.831327 27344 consensus_queue.cc:260] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:11.835702 27086 heartbeater.cc:503] Master 127.16.86.190:43733 was elected leader, sending a full tablet report...
I20260806 09:22:11.835898 27343 ts_tablet_manager.cc:1428] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa: Time spent starting tablet: real 0.023s user 0.021s sys 0.001s
I20260806 09:22:11.838460 27290 heartbeater.cc:503] Master 127.16.86.190:43733 was elected leader, sending a full tablet report...
I20260806 09:22:11.841195 27344 ts_tablet_manager.cc:1428] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502: Time spent starting tablet: real 0.025s user 0.010s sys 0.004s
W20260806 09:22:11.875962 27293 tablet.cc:2369] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:22:11.889524 26888 tablet.cc:2369] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:22:11.902034 27350 raft_consensus.cc:482] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:22:11.902410 27350 raft_consensus.cc:504] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:11.904606 27350 leader_election.cc:288] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3d9877aba6a445280ab89df34417458 (127.16.86.129:45867), 5cb7873ac0eb467ebce54384aee30caa (127.16.86.130:33775)
I20260806 09:22:11.914646 27339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.905717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:37381 (local address 127.16.86.130:33775)
0806 09:22:11.905968 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:11.905975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:11.906002 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:22:11.906104 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:11.906109 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:11.906189 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:11.906338 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:11.907053 (+ 715us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.908416 (+ 1363us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.911839 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.913297 (+ 1458us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.913532 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.913619 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:11.914177 (+ 558us) server_negotiation.cc:299] Negotiation successful
0806 09:22:11.914329 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 09:22:11.914700 27338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.905759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:54379 (local address 127.16.86.129:45867)
0806 09:22:11.905968 (+ 209us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:11.905975 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:11.906002 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:22:11.906091 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:11.906097 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:11.906189 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:11.906338 (+ 149us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:11.907062 (+ 724us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.908416 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.911839 (+ 3423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.913382 (+ 1543us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.913551 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.913634 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:11.914177 (+ 543us) server_negotiation.cc:299] Negotiation successful
0806 09:22:11.914329 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20260806 09:22:11.914819 27299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.905074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:45867 (local address 127.16.86.131:54379)
0806 09:22:11.905566 (+ 492us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:11.905594 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:11.905767 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:11.906383 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:11.906388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:11.906433 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:11.906811 (+ 378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.906825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.908642 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.908648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.911612 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.911633 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.913576 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.913583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.913926 (+ 343us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.913990 (+ 64us) client_negotiation.cc:769] Sending connection context
0806 09:22:11.914109 (+ 119us) client_negotiation.cc:240] Negotiation successful
0806 09:22:11.914356 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":265,"mutex_wait_us":489,"spinlock_wait_cycles":59648}
I20260806 09:22:11.915545 27298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.905271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:33775 (local address 127.16.86.131:37381)
0806 09:22:11.905589 (+ 318us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:11.905604 (+ 15us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:11.905767 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:11.906383 (+ 616us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:11.906388 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:11.906418 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:11.906811 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.906825 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.908642 (+ 1817us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.908648 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.911612 (+ 2964us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:11.911633 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.913576 (+ 1943us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.913583 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:11.913993 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.914018 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:22:11.914109 (+ 91us) client_negotiation.cc:240] Negotiation successful
0806 09:22:11.914356 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":123,"mutex_wait_us":477,"spinlock_wait_cycles":52992}
I20260806 09:22:11.916098 26821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a249d912b7d45ecab79d81bfbe347aa" candidate_uuid: "d8f51643bf134d19843fc65d7fd39502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d9877aba6a445280ab89df34417458" is_pre_election: true
I20260806 09:22:11.916734 26821 raft_consensus.cc:2454] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f51643bf134d19843fc65d7fd39502 in term 0.
I20260806 09:22:11.916992 27025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a249d912b7d45ecab79d81bfbe347aa" candidate_uuid: "d8f51643bf134d19843fc65d7fd39502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb7873ac0eb467ebce54384aee30caa" is_pre_election: true
I20260806 09:22:11.917682 27025 raft_consensus.cc:2454] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d8f51643bf134d19843fc65d7fd39502 in term 0.
I20260806 09:22:11.917637 27125 leader_election.cc:302] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3d9877aba6a445280ab89df34417458, d8f51643bf134d19843fc65d7fd39502; no voters:
I20260806 09:22:11.918221 27350 raft_consensus.cc:2781] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:22:11.918490 27350 raft_consensus.cc:482] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:22:11.918634 27350 raft_consensus.cc:3037] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:11.922175 27350 raft_consensus.cc:504] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:11.923063 27350 leader_election.cc:288] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [CANDIDATE]: Term 1 election: Requested vote from peers d3d9877aba6a445280ab89df34417458 (127.16.86.129:45867), 5cb7873ac0eb467ebce54384aee30caa (127.16.86.130:33775)
I20260806 09:22:11.923673 26821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a249d912b7d45ecab79d81bfbe347aa" candidate_uuid: "d8f51643bf134d19843fc65d7fd39502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d9877aba6a445280ab89df34417458"
I20260806 09:22:11.923673 27025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a249d912b7d45ecab79d81bfbe347aa" candidate_uuid: "d8f51643bf134d19843fc65d7fd39502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb7873ac0eb467ebce54384aee30caa"
I20260806 09:22:11.924008 27025 raft_consensus.cc:3037] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:11.923979 26821 raft_consensus.cc:3037] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:11.927896 26821 raft_consensus.cc:2454] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f51643bf134d19843fc65d7fd39502 in term 1.
I20260806 09:22:11.927896 27025 raft_consensus.cc:2454] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d8f51643bf134d19843fc65d7fd39502 in term 1.
I20260806 09:22:11.928589 27126 leader_election.cc:302] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 5cb7873ac0eb467ebce54384aee30caa, d8f51643bf134d19843fc65d7fd39502; no voters:
I20260806 09:22:11.929045 27350 raft_consensus.cc:2781] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:11.929739 27350 raft_consensus.cc:686] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 1 LEADER]: Becoming Leader. State: Replica: d8f51643bf134d19843fc65d7fd39502, State: Running, Role: LEADER
I20260806 09:22:11.930759 27350 consensus_queue.cc:237] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:11.937888 25460 catalog_manager.cc:5495] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 reported cstate change: term changed from 0 to 1, leader changed from <none> to d8f51643bf134d19843fc65d7fd39502 (127.16.86.131). New cstate: current_term: 1 leader_uuid: "d8f51643bf134d19843fc65d7fd39502" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } health_report { overall_health: UNKNOWN } } }
I20260806 09:22:11.975677 27334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:11.969488 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.16.86.131:36101)
0806 09:22:11.969787 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:11.969795 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:11.969825 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:22:11.969944 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:11.969948 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:11.970049 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:11.970229 (+ 180us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:22:11.970670 (+ 441us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.971592 (+ 922us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.973110 (+ 1518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:11.973546 (+ 436us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:11.973696 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:11.974277 (+ 581us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:22:11.974931 (+ 654us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:22:11.975200 (+ 269us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:11.975255 (+ 55us) server_negotiation.cc:299] Negotiation successful
0806 09:22:11.975353 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 09:22:11.986523 27025 raft_consensus.cc:1261] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa [term 1 FOLLOWER]: Refusing update from remote peer d8f51643bf134d19843fc65d7fd39502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:11.986515 26821 raft_consensus.cc:1261] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458 [term 1 FOLLOWER]: Refusing update from remote peer d8f51643bf134d19843fc65d7fd39502: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:11.987895 27354 consensus_queue.cc:1035] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:11.988387 27350 consensus_queue.cc:1035] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:12.007194 27360 mvcc.cc:205] Tried to move back new op lower bound from 7315489308602695680 to 7315489308393762816. Current Snapshot: MvccSnapshot[applied={T|T < 7315489308602695680}]
I20260806 09:22:12.025272 25460 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:12.025625 25460 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53826:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:12.034607 25460 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 4b2fc5f140794fee87c356b29cd9fb6a: Sending DeleteTablet for 3 replicas of tablet 1a249d912b7d45ecab79d81bfbe347aa
I20260806 09:22:12.036316 27189 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a249d912b7d45ecab79d81bfbe347aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:12 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:38942
I20260806 09:22:12.036623 26999 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a249d912b7d45ecab79d81bfbe347aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:12 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:51456
I20260806 09:22:12.038057 26792 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a249d912b7d45ecab79d81bfbe347aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:12 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:52380
I20260806 09:22:12.047823 27373 tablet_replica.cc:330] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458: stopping tablet replica
I20260806 09:22:12.048933 27373 raft_consensus.cc:2229] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:12.051044 27373 raft_consensus.cc:2258] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:12.052435 27374 tablet_replica.cc:330] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502: stopping tablet replica
I20260806 09:22:12.053339 27374 raft_consensus.cc:2229] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:22:12.055465 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 25413
I20260806 09:22:12.056565 27373 ts_tablet_manager.cc:1905] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:12.056947 25508 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:12.055627 27374 raft_consensus.cc:2258] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:12.062994 27374 ts_tablet_manager.cc:1905] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:12.064020 27370 tablet_replica.cc:330] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa: stopping tablet replica
I20260806 09:22:12.066368 27370 raft_consensus.cc:2229] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:12.069734 27370 raft_consensus.cc:2258] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:12.076723 27370 ts_tablet_manager.cc:1905] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:12.079525 27374 ts_tablet_manager.cc:1918] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:12.079782 27374 log.cc:1196] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/1a249d912b7d45ecab79d81bfbe347aa
I20260806 09:22:12.080601 27374 ts_tablet_manager.cc:1939] T 1a249d912b7d45ecab79d81bfbe347aa P d8f51643bf134d19843fc65d7fd39502: Deleting consensus metadata
I20260806 09:22:12.084266 27373 ts_tablet_manager.cc:1918] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:12.084523 27373 log.cc:1196] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/1a249d912b7d45ecab79d81bfbe347aa
I20260806 09:22:12.085227 27373 ts_tablet_manager.cc:1939] T 1a249d912b7d45ecab79d81bfbe347aa P d3d9877aba6a445280ab89df34417458: Deleting consensus metadata
I20260806 09:22:12.090415 27370 ts_tablet_manager.cc:1918] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:12.090668 27370 log.cc:1196] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/1a249d912b7d45ecab79d81bfbe347aa
I20260806 09:22:12.091444 27370 ts_tablet_manager.cc:1939] T 1a249d912b7d45ecab79d81bfbe347aa P 5cb7873ac0eb467ebce54384aee30caa: Deleting consensus metadata
I20260806 09:22:12.094267 27311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:12.044394 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53160 (local address 127.16.86.189:46587)
0806 09:22:12.044629 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:12.044636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:12.056025 (+ 11389us) server_negotiation.cc:407] Connection header received
0806 09:22:12.056433 (+ 408us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:12.056438 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:12.056549 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:12.056644 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:12.060695 (+ 4051us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:12.061929 (+ 1234us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:12.067276 (+ 5347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:12.068026 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:12.068153 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:12.078953 (+ 10800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:12.078965 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:12.078974 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:12.079067 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:12.083468 (+ 4401us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:12.090976 (+ 7508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:12.090985 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:12.090989 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:12.091313 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:12.092114 (+ 801us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:12.092121 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:12.092127 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:12.092581 (+ 454us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:12.092897 (+ 316us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:12.093636 (+ 739us) server_negotiation.cc:299] Negotiation successful
0806 09:22:12.093979 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:22:12.115085 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:12.054501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51112 (local address 127.16.86.188:33963)
0806 09:22:12.054699 (+ 198us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:12.054706 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:12.095456 (+ 40750us) server_negotiation.cc:407] Connection header received
0806 09:22:12.095734 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:12.095743 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:12.095869 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:12.095981 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:12.098464 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:12.099843 (+ 1379us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:12.101912 (+ 2069us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:12.102680 (+ 768us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:12.102851 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:12.106895 (+ 4044us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:12.106903 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:12.106909 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:12.106964 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:12.111318 (+ 4354us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:12.112373 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:12.112380 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:12.112385 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:12.112583 (+ 198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:12.113104 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:12.113110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:12.113114 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:12.113611 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:12.113901 (+ 290us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:12.114339 (+ 438us) server_negotiation.cc:299] Negotiation successful
0806 09:22:12.114621 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20260806 09:22:12.229238 25444 catalog_manager.cc:4841] TS d8f51643bf134d19843fc65d7fd39502 (127.16.86.131:36101): tablet 1a249d912b7d45ecab79d81bfbe347aa (table test-table [id=e9120faf946a4ae29f1c90a9ad6fc78d]) successfully deleted
I20260806 09:22:12.229447 25442 catalog_manager.cc:4841] TS 5cb7873ac0eb467ebce54384aee30caa (127.16.86.130:33775): tablet 1a249d912b7d45ecab79d81bfbe347aa (table test-table [id=e9120faf946a4ae29f1c90a9ad6fc78d]) successfully deleted
I20260806 09:22:12.229496 25441 catalog_manager.cc:4841] TS d3d9877aba6a445280ab89df34417458 (127.16.86.129:45867): tablet 1a249d912b7d45ecab79d81bfbe347aa (table test-table [id=e9120faf946a4ae29f1c90a9ad6fc78d]) successfully deleted
I20260806 09:22:12.230436 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 25413
I20260806 09:22:12.291811 27311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:12.273167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53162 (local address 127.16.86.189:46587)
0806 09:22:12.273454 (+ 287us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:12.273464 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:12.274735 (+ 1271us) server_negotiation.cc:407] Connection header received
0806 09:22:12.275132 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:12.275138 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:12.275249 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:12.275381 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:12.277846 (+ 2465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:12.279381 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:12.281086 (+ 1705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:12.281577 (+ 491us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:12.281734 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:12.285371 (+ 3637us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:12.285377 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:12.285381 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:12.285431 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:12.288565 (+ 3134us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:12.289506 (+ 941us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:12.289512 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:12.289514 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:12.289657 (+ 143us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:12.290049 (+ 392us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:12.290056 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:12.290059 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:12.290400 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:12.290689 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:12.291074 (+ 385us) server_negotiation.cc:299] Negotiation successful
0806 09:22:12.291350 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:22:12.294790 27295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:12.274906 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51120 (local address 127.16.86.188:33963)
0806 09:22:12.275163 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:12.275169 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:12.275191 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:22:12.275368 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:12.275373 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:12.275467 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:12.275581 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:12.278430 (+ 2849us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:12.279590 (+ 1160us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:12.282077 (+ 2487us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:12.282693 (+ 616us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:12.282855 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:12.286972 (+ 4117us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:12.286982 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:12.286987 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:12.287101 (+ 114us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:12.290989 (+ 3888us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:12.292000 (+ 1011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:12.292006 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:12.292011 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:12.292188 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:12.292695 (+ 507us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:12.292701 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:12.292704 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:12.293195 (+ 491us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:12.293527 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:12.294100 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 09:22:12.294399 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260806 09:22:13.013377 27388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.012129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.131:53179)
0806 09:22:13.012778 (+ 649us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.013079 (+ 301us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":324,"threads_started":1}
W20260806 09:22:13.014303 27290 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:43733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
I20260806 09:22:13.019641 27389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.018612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.130:60607)
0806 09:22:13.019249 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.019427 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":328,"threads_started":1}
I20260806 09:22:13.020031 27390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.019060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.129:44481)
0806 09:22:13.019598 (+ 538us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.019788 (+ 190us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":299,"threads_started":1}
W20260806 09:22:13.020108 27086 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:43733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
W20260806 09:22:13.020510 26886 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:43733 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
I20260806 09:22:13.535218 27391 raft_consensus.cc:482] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b2fc5f140794fee87c356b29cd9fb6a)
I20260806 09:22:13.535632 27391 raft_consensus.cc:504] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } }
I20260806 09:22:13.535741 27392 raft_consensus.cc:482] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4b2fc5f140794fee87c356b29cd9fb6a)
I20260806 09:22:13.536099 27392 raft_consensus.cc:504] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } }
I20260806 09:22:13.536868 27391 leader_election.cc:288] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b2fc5f140794fee87c356b29cd9fb6a (127.16.86.190:43733), fd6d8f8101f447c7a37af25d89cceaf9 (127.16.86.189:46587)
I20260806 09:22:13.537859 27392 leader_election.cc:288] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4b2fc5f140794fee87c356b29cd9fb6a (127.16.86.190:43733), dee9589594ac4444bff7783dc8b59f3b (127.16.86.188:33963)
I20260806 09:22:13.539592 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.537142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:54252)
0806 09:22:13.538933 (+ 1791us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.539268 (+ 335us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1639,"threads_started":1}
I20260806 09:22:13.540412 27399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.537981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:54254)
0806 09:22:13.539579 (+ 1598us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.540150 (+ 571us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1446,"thread_start_us":478,"threads_started":1}
W20260806 09:22:13.540858 26312 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
W20260806 09:22:13.541983 26603 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
I20260806 09:22:13.550145 27397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.537325 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53168 (local address 127.16.86.189:46587)
0806 09:22:13.540001 (+ 2676us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:13.540010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:13.540107 (+ 97us) server_negotiation.cc:407] Connection header received
0806 09:22:13.540433 (+ 326us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:13.540440 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:13.540526 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:13.540615 (+ 89us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:13.541568 (+ 953us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.542968 (+ 1400us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.546422 (+ 3454us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.548314 (+ 1892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.548461 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.548620 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:13.549377 (+ 757us) server_negotiation.cc:299] Negotiation successful
0806 09:22:13.549593 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2554,"thread_start_us":7440,"threads_started":1}
I20260806 09:22:13.551293 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.537278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46587 (local address 127.0.0.1:53168)
0806 09:22:13.539993 (+ 2715us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.540013 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:13.540234 (+ 221us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:13.540974 (+ 740us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:13.540978 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:13.541011 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:13.541295 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.541314 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.543249 (+ 1935us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.543257 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.546213 (+ 2956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.546231 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.548846 (+ 2615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.548856 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.549128 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.549184 (+ 56us) client_negotiation.cc:769] Sending connection context
0806 09:22:13.550561 (+ 1377us) client_negotiation.cc:240] Negotiation successful
0806 09:22:13.550802 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2601,"thread_start_us":1329,"threads_started":1}
I20260806 09:22:13.552665 26342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dee9589594ac4444bff7783dc8b59f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" is_pre_election: true
I20260806 09:22:13.553123 26342 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate dee9589594ac4444bff7783dc8b59f3b in term 1.
I20260806 09:22:13.554091 26601 leader_election.cc:302] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: dee9589594ac4444bff7783dc8b59f3b, fd6d8f8101f447c7a37af25d89cceaf9; no voters:
I20260806 09:22:13.554804 27391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260806 09:22:13.555105 27391 raft_consensus.cc:482] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4b2fc5f140794fee87c356b29cd9fb6a)
I20260806 09:22:13.555367 27391 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:22:13.559082 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.558541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:54280)
0806 09:22:13.558716 (+ 175us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.558818 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20260806 09:22:13.559450 26603 leader_election.cc:334] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b2fc5f140794fee87c356b29cd9fb6a (127.16.86.190:43733): Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
I20260806 09:22:13.560015 27407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.552885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:54270)
0806 09:22:13.558933 (+ 6048us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.559461 (+ 528us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5846,"thread_start_us":5671,"threads_started":1}
W20260806 09:22:13.560417 26312 leader_election.cc:334] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer 4b2fc5f140794fee87c356b29cd9fb6a (127.16.86.190:43733): Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
I20260806 09:22:13.561017 27399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.539337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:33963 (local address 127.0.0.1:51132)
0806 09:22:13.540975 (+ 1638us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.541003 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:13.541101 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:13.552729 (+ 11628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:13.552737 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:13.552828 (+ 91us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:13.553252 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.553268 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.556111 (+ 2843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.556119 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.558364 (+ 2245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.558381 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.560381 (+ 2000us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.560387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.560526 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.560554 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 09:22:13.560707 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 09:22:13.560826 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1541,"mutex_wait_us":1306}
I20260806 09:22:13.561811 27400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.538899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51132 (local address 127.16.86.188:33963)
0806 09:22:13.552249 (+ 13350us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:13.552258 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:13.552282 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:22:13.552362 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:13.552370 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:13.552480 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:13.553501 (+ 1021us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:13.553817 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.555888 (+ 2071us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.558578 (+ 2690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.560080 (+ 1502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.560904 (+ 824us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.561088 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:13.561181 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 09:22:13.561393 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13156,"thread_start_us":12972,"threads_started":1}
I20260806 09:22:13.562688 26636 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "dee9589594ac4444bff7783dc8b59f3b" is_pre_election: true
I20260806 09:22:13.562784 27391 raft_consensus.cc:504] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } }
I20260806 09:22:13.563504 26636 raft_consensus.cc:2379] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd6d8f8101f447c7a37af25d89cceaf9 in current term 2: Already voted for candidate dee9589594ac4444bff7783dc8b59f3b in this term.
I20260806 09:22:13.563951 27391 leader_election.cc:288] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [CANDIDATE]: Term 2 election: Requested vote from peers 4b2fc5f140794fee87c356b29cd9fb6a (127.16.86.190:43733), fd6d8f8101f447c7a37af25d89cceaf9 (127.16.86.189:46587)
I20260806 09:22:13.564441 26308 leader_election.cc:302] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd6d8f8101f447c7a37af25d89cceaf9; no voters: 4b2fc5f140794fee87c356b29cd9fb6a, dee9589594ac4444bff7783dc8b59f3b
I20260806 09:22:13.564469 26342 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "dee9589594ac4444bff7783dc8b59f3b" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "fd6d8f8101f447c7a37af25d89cceaf9"
I20260806 09:22:13.564738 26342 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:22:13.564936 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.564089 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:54292)
0806 09:22:13.564604 (+ 515us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.564741 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317}
I20260806 09:22:13.566268 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.565759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:54308)
0806 09:22:13.565969 (+ 210us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.566086 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
W20260806 09:22:13.566509 26603 leader_election.cc:334] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer 4b2fc5f140794fee87c356b29cd9fb6a (127.16.86.190:43733): Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
I20260806 09:22:13.567589 26342 raft_consensus.cc:2454] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate dee9589594ac4444bff7783dc8b59f3b in term 2.
I20260806 09:22:13.567788 27392 raft_consensus.cc:2726] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 09:22:13.568114 26601 leader_election.cc:302] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: dee9589594ac4444bff7783dc8b59f3b, fd6d8f8101f447c7a37af25d89cceaf9; no voters: 4b2fc5f140794fee87c356b29cd9fb6a
I20260806 09:22:13.568535 27391 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:22:13.569135 27391 raft_consensus.cc:686] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [term 2 LEADER]: Becoming Leader. State: Replica: dee9589594ac4444bff7783dc8b59f3b, State: Running, Role: LEADER
I20260806 09:22:13.569558 27391 consensus_queue.cc:237] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } }
I20260806 09:22:13.571671 27408 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: SysCatalogTable state changed. Reason: New leader dee9589594ac4444bff7783dc8b59f3b. Latest consensus state: current_term: 2 leader_uuid: "dee9589594ac4444bff7783dc8b59f3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } } }
I20260806 09:22:13.572019 27408 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:13.572882 27410 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:22:13.578948 27410 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:22:13.580376 27410 catalog_manager.cc:1259] Loaded cluster ID: 4c0afa372dac4077b1e53622c7454eba
I20260806 09:22:13.580521 27410 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:22:13.582019 27410 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:22:13.583221 27410 catalog_manager.cc:5879] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: Loaded TSK: 0
I20260806 09:22:13.584429 27410 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:22:13.603718 26619 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51120:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:22:13.613907 26619 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:22:13.619630 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.619086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:54340)
0806 09:22:13.619291 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.619473 (+ 182us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":60}
I20260806 09:22:13.619859 26342 raft_consensus.cc:1261] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [term 2 FOLLOWER]: Refusing update from remote peer dee9589594ac4444bff7783dc8b59f3b: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 2 index: 14. (index mismatch)
I20260806 09:22:13.620873 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.620480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:54348)
0806 09:22:13.620677 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.620751 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 09:22:13.620802 27408 consensus_queue.cc:1035] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20260806 09:22:13.621129 26603 consensus_peers.cc:487] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b -> Peer 4b2fc5f140794fee87c356b29cd9fb6a (127.16.86.190:43733): Couldn't send request to peer 4b2fc5f140794fee87c356b29cd9fb6a. Status: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260806 09:22:13.624051 27392 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dee9589594ac4444bff7783dc8b59f3b. Latest consensus state: current_term: 2 leader_uuid: "dee9589594ac4444bff7783dc8b59f3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } } }
I20260806 09:22:13.624356 27392 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:22:13.626169 27408 sys_catalog.cc:455] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "dee9589594ac4444bff7783dc8b59f3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } } }
I20260806 09:22:13.626554 27408 sys_catalog.cc:458] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:13.627082 27392 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "dee9589594ac4444bff7783dc8b59f3b" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "4b2fc5f140794fee87c356b29cd9fb6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43733 } } peers { permanent_uuid: "fd6d8f8101f447c7a37af25d89cceaf9" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46587 } attrs { promote: false } } peers { permanent_uuid: "dee9589594ac4444bff7783dc8b59f3b" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 33963 } attrs { promote: false } } }
I20260806 09:22:13.627493 27392 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd6d8f8101f447c7a37af25d89cceaf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:22:13.646828 27395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.638100 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36101 (local address 127.0.0.1:38960)
0806 09:22:13.638547 (+ 447us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.638571 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:13.638704 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:13.639186 (+ 482us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:13.639194 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:13.639229 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:13.639882 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.639898 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.641435 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.641442 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.643243 (+ 1801us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.643256 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.645132 (+ 1876us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.645145 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.645653 (+ 508us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.645726 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 09:22:13.645901 (+ 175us) client_negotiation.cc:240] Negotiation successful
0806 09:22:13.646178 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":258,"mutex_wait_us":62}
I20260806 09:22:13.647969 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.638182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38960 (local address 127.16.86.131:36101)
0806 09:22:13.638663 (+ 481us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:13.638669 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:13.638687 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:22:13.638868 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:13.638874 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:13.638966 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:13.640223 (+ 1257us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:13.640449 (+ 226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.641299 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.643445 (+ 2146us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.644728 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.644874 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.645012 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:13.647416 (+ 2404us) server_negotiation.cc:299] Negotiation successful
0806 09:22:13.647582 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":348,"thread_start_us":661,"threads_started":1}
I20260806 09:22:13.648089 27396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.637603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:45867 (local address 127.0.0.1:52388)
0806 09:22:13.637919 (+ 316us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.637946 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:13.638086 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:13.640788 (+ 2702us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:13.640797 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:13.640838 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:13.641232 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.641249 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.643612 (+ 2363us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.643618 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.645516 (+ 1898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.645528 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.647266 (+ 1738us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.647272 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.647484 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.647525 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:22:13.647595 (+ 70us) client_negotiation.cc:240] Negotiation successful
0806 09:22:13.647743 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":135,"mutex_wait_us":85}
I20260806 09:22:13.648969 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.637709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52388 (local address 127.16.86.129:45867)
0806 09:22:13.640288 (+ 2579us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:13.640295 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:13.640318 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:22:13.640416 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:13.640425 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:13.640549 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:13.641430 (+ 881us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:13.641661 (+ 231us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.643426 (+ 1765us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.645727 (+ 2301us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.647071 (+ 1344us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.648319 (+ 1248us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.648430 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:13.648490 (+ 60us) server_negotiation.cc:299] Negotiation successful
0806 09:22:13.648650 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2397,"thread_start_us":2260,"threads_started":1}
I20260806 09:22:13.649032 27189 tablet_service.cc:1467] Processing CreateTablet for tablet a937fa3212f949c2b488406d08d93a5d (DEFAULT_TABLE table=test-table [id=452c07ba43334704a47cdc36a3f25306]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:13.649621 27189 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a937fa3212f949c2b488406d08d93a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:13.650019 26792 tablet_service.cc:1467] Processing CreateTablet for tablet a937fa3212f949c2b488406d08d93a5d (DEFAULT_TABLE table=test-table [id=452c07ba43334704a47cdc36a3f25306]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:13.650650 26792 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a937fa3212f949c2b488406d08d93a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:13.654917 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.638709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:33775 (local address 127.0.0.1:51470)
0806 09:22:13.640520 (+ 1811us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.640542 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:13.640659 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:13.648583 (+ 7924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:13.648590 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:13.648622 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:13.648926 (+ 304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.648935 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.650904 (+ 1969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.650911 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.652802 (+ 1891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.652815 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.654084 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.654090 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.654265 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.654291 (+ 26us) client_negotiation.cc:769] Sending connection context
0806 09:22:13.654407 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 09:22:13.654549 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1677,"thread_start_us":1520,"threads_started":1}
I20260806 09:22:13.654906 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.639256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51470 (local address 127.16.86.130:33775)
0806 09:22:13.647971 (+ 8715us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:13.647979 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:13.648009 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:22:13.648120 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:13.648128 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:13.648250 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:13.648372 (+ 122us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:13.649351 (+ 979us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.650584 (+ 1233us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.653032 (+ 2448us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.653850 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.654023 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.654145 (+ 122us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:13.654436 (+ 291us) server_negotiation.cc:299] Negotiation successful
0806 09:22:13.654579 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8525,"thread_start_us":8360,"threads_started":1}
I20260806 09:22:13.659809 27428 tablet_bootstrap.cc:492] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502: Bootstrap starting.
I20260806 09:22:13.660056 26999 tablet_service.cc:1467] Processing CreateTablet for tablet a937fa3212f949c2b488406d08d93a5d (DEFAULT_TABLE table=test-table [id=452c07ba43334704a47cdc36a3f25306]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:13.660498 27427 tablet_bootstrap.cc:492] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458: Bootstrap starting.
I20260806 09:22:13.660743 26999 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet a937fa3212f949c2b488406d08d93a5d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:13.661871 27428 tablet_bootstrap.cc:654] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:13.662851 27427 tablet_bootstrap.cc:654] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:13.665210 27428 tablet_bootstrap.cc:492] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502: No bootstrap required, opened a new log
I20260806 09:22:13.665383 27428 ts_tablet_manager.cc:1397] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260806 09:22:13.666232 27428 raft_consensus.cc:348] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:13.666468 27428 raft_consensus.cc:374] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:13.666558 27428 raft_consensus.cc:729] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f51643bf134d19843fc65d7fd39502, State: Initialized, Role: FOLLOWER
I20260806 09:22:13.666814 27428 consensus_queue.cc:260] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:13.668753 27428 ts_tablet_manager.cc:1428] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 09:22:13.673373 27289 heartbeater.cc:503] Master 127.16.86.188:33963 was elected leader, sending a full tablet report...
I20260806 09:22:13.673872 27434 tablet_bootstrap.cc:492] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa: Bootstrap starting.
I20260806 09:22:13.673986 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.668168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.131:47653)
0806 09:22:13.673228 (+ 5060us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.673576 (+ 348us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":4849,"thread_start_us":4618,"threads_started":1}
I20260806 09:22:13.675998 27433 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.674771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.131:57169)
0806 09:22:13.675710 (+ 939us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.675814 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":815,"threads_started":1}
I20260806 09:22:13.678115 27434 tablet_bootstrap.cc:654] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:13.678139 27432 raft_consensus.cc:482] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:22:13.678656 27432 raft_consensus.cc:504] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:13.680207 27432 leader_election.cc:288] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d3d9877aba6a445280ab89df34417458 (127.16.86.129:45867), 5cb7873ac0eb467ebce54384aee30caa (127.16.86.130:33775)
I20260806 09:22:13.680507 26821 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a937fa3212f949c2b488406d08d93a5d" candidate_uuid: "d8f51643bf134d19843fc65d7fd39502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d3d9877aba6a445280ab89df34417458" is_pre_election: true
I20260806 09:22:13.680984 27025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a937fa3212f949c2b488406d08d93a5d" candidate_uuid: "d8f51643bf134d19843fc65d7fd39502" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb7873ac0eb467ebce54384aee30caa" is_pre_election: true
I20260806 09:22:13.681581 27427 tablet_bootstrap.cc:492] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458: No bootstrap required, opened a new log
W20260806 09:22:13.681573 27125 leader_election.cc:341] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d3d9877aba6a445280ab89df34417458 (127.16.86.129:45867): Illegal state: must be running to vote when last-logged opid is not known
I20260806 09:22:13.681730 27427 ts_tablet_manager.cc:1397] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.007s
W20260806 09:22:13.682227 27126 leader_election.cc:341] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 5cb7873ac0eb467ebce54384aee30caa (127.16.86.130:33775): Illegal state: must be running to vote when last-logged opid is not known
I20260806 09:22:13.682497 27126 leader_election.cc:302] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d8f51643bf134d19843fc65d7fd39502; no voters: 5cb7873ac0eb467ebce54384aee30caa, d3d9877aba6a445280ab89df34417458
I20260806 09:22:13.682951 27427 raft_consensus.cc:348] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:13.683238 27427 raft_consensus.cc:374] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:13.683220 27432 raft_consensus.cc:2726] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 09:22:13.683333 27427 raft_consensus.cc:729] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d3d9877aba6a445280ab89df34417458, State: Initialized, Role: FOLLOWER
I20260806 09:22:13.683619 27427 consensus_queue.cc:260] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:13.684948 27427 ts_tablet_manager.cc:1428] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260806 09:22:13.688433 27434 tablet_bootstrap.cc:492] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa: No bootstrap required, opened a new log
I20260806 09:22:13.688618 27434 ts_tablet_manager.cc:1397] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.004s
I20260806 09:22:13.689085 26885 heartbeater.cc:503] Master 127.16.86.188:33963 was elected leader, sending a full tablet report...
I20260806 09:22:13.689581 27434 raft_consensus.cc:348] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:13.689875 27434 raft_consensus.cc:374] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:13.689981 27434 raft_consensus.cc:729] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cb7873ac0eb467ebce54384aee30caa, State: Initialized, Role: FOLLOWER
I20260806 09:22:13.690184 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.686173 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.129:56139)
0806 09:22:13.688712 (+ 2539us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.689010 (+ 298us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2357,"thread_start_us":2126,"threads_started":1}
I20260806 09:22:13.690251 27434 consensus_queue.cc:260] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:13.691695 27434 ts_tablet_manager.cc:1428] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa: Time spent starting tablet: real 0.003s user 0.000s sys 0.003s
I20260806 09:22:13.692271 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.691678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.129:60887)
0806 09:22:13.691983 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.692089 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260806 09:22:13.693944 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.692190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.130:40793)
0806 09:22:13.693456 (+ 1266us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.693659 (+ 203us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1131,"thread_start_us":875,"threads_started":1}
I20260806 09:22:13.694809 27084 heartbeater.cc:503] Master 127.16.86.188:33963 was elected leader, sending a full tablet report...
I20260806 09:22:13.695658 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.695306 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.130:40313)
0806 09:22:13.695462 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.695548 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 09:22:13.721065 27438 raft_consensus.cc:482] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:22:13.721323 27438 raft_consensus.cc:504] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:13.722944 27438 leader_election.cc:288] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d8f51643bf134d19843fc65d7fd39502 (127.16.86.131:36101), 5cb7873ac0eb467ebce54384aee30caa (127.16.86.130:33775)
I20260806 09:22:13.729764 27439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.723202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:36101 (local address 127.16.86.129:39235)
0806 09:22:13.723476 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.723501 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:13.723705 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:13.724306 (+ 601us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:13.724318 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:13.724344 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:13.724833 (+ 489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.724854 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.726156 (+ 1302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.726164 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.727721 (+ 1557us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.727735 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.728946 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.728952 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.729144 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.729169 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:22:13.729296 (+ 127us) client_negotiation.cc:240] Negotiation successful
0806 09:22:13.729460 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":56}
I20260806 09:22:13.729755 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.723522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:39235 (local address 127.16.86.131:36101)
0806 09:22:13.723817 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:13.723824 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:13.723844 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:22:13.723924 (+ 80us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:13.723929 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:13.724025 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:13.724123 (+ 98us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:13.725025 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.726022 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.727885 (+ 1863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.728705 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.728921 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.728999 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:13.729326 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 09:22:13.729460 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:22:13.731105 27216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a937fa3212f949c2b488406d08d93a5d" candidate_uuid: "d3d9877aba6a445280ab89df34417458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f51643bf134d19843fc65d7fd39502" is_pre_election: true
I20260806 09:22:13.731556 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.723670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:33775 (local address 127.16.86.129:51209)
0806 09:22:13.724552 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.724572 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:13.724828 (+ 256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:13.725442 (+ 614us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:13.725447 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:13.725475 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:13.725983 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.725998 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.727342 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.727357 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.728984 (+ 1627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:13.728999 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.730550 (+ 1551us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.730555 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:13.730828 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.730877 (+ 49us) client_negotiation.cc:769] Sending connection context
0806 09:22:13.730995 (+ 118us) client_negotiation.cc:240] Negotiation successful
0806 09:22:13.731194 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":501,"threads_started":1}
I20260806 09:22:13.731647 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.723892 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:51209 (local address 127.16.86.130:33775)
0806 09:22:13.724176 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:13.724182 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:13.725078 (+ 896us) server_negotiation.cc:407] Connection header received
0806 09:22:13.725191 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:13.725196 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:13.725271 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:13.725395 (+ 124us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:13.726246 (+ 851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.727178 (+ 932us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.729185 (+ 2007us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.730296 (+ 1111us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.730518 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.730616 (+ 98us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:13.731148 (+ 532us) server_negotiation.cc:299] Negotiation successful
0806 09:22:13.731296 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 09:22:13.731647 27216 raft_consensus.cc:2454] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3d9877aba6a445280ab89df34417458 in term 0.
I20260806 09:22:13.732328 26732 leader_election.cc:302] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d3d9877aba6a445280ab89df34417458, d8f51643bf134d19843fc65d7fd39502; no voters:
I20260806 09:22:13.732767 27438 raft_consensus.cc:2781] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:22:13.732741 27025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a937fa3212f949c2b488406d08d93a5d" candidate_uuid: "d3d9877aba6a445280ab89df34417458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb7873ac0eb467ebce54384aee30caa" is_pre_election: true
I20260806 09:22:13.732913 27438 raft_consensus.cc:482] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:22:13.733038 27438 raft_consensus.cc:3037] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:13.733148 27025 raft_consensus.cc:2454] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d3d9877aba6a445280ab89df34417458 in term 0.
I20260806 09:22:13.735854 27438 raft_consensus.cc:504] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:13.736639 27438 leader_election.cc:288] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [CANDIDATE]: Term 1 election: Requested vote from peers d8f51643bf134d19843fc65d7fd39502 (127.16.86.131:36101), 5cb7873ac0eb467ebce54384aee30caa (127.16.86.130:33775)
I20260806 09:22:13.737413 27216 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a937fa3212f949c2b488406d08d93a5d" candidate_uuid: "d3d9877aba6a445280ab89df34417458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d8f51643bf134d19843fc65d7fd39502"
I20260806 09:22:13.737656 27216 raft_consensus.cc:3037] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:13.737542 27025 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "a937fa3212f949c2b488406d08d93a5d" candidate_uuid: "d3d9877aba6a445280ab89df34417458" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cb7873ac0eb467ebce54384aee30caa"
I20260806 09:22:13.737946 27025 raft_consensus.cc:3037] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:13.740612 27216 raft_consensus.cc:2454] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3d9877aba6a445280ab89df34417458 in term 1.
I20260806 09:22:13.740749 27025 raft_consensus.cc:2454] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d3d9877aba6a445280ab89df34417458 in term 1.
I20260806 09:22:13.741425 26732 leader_election.cc:302] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 5cb7873ac0eb467ebce54384aee30caa, d3d9877aba6a445280ab89df34417458, d8f51643bf134d19843fc65d7fd39502; no voters:
I20260806 09:22:13.741902 27438 raft_consensus.cc:2781] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:13.742542 27438 raft_consensus.cc:686] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 1 LEADER]: Becoming Leader. State: Replica: d3d9877aba6a445280ab89df34417458, State: Running, Role: LEADER
I20260806 09:22:13.743099 27438 consensus_queue.cc:237] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } } peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } }
I20260806 09:22:13.744133 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.743466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.129:43169)
0806 09:22:13.743772 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.743925 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":165}
I20260806 09:22:13.747856 26619 catalog_manager.cc:5495] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 reported cstate change: term changed from 0 to 1, leader changed from <none> to d3d9877aba6a445280ab89df34417458 (127.16.86.129). New cstate: current_term: 1 leader_uuid: "d3d9877aba6a445280ab89df34417458" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d3d9877aba6a445280ab89df34417458" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45867 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 } health_report { overall_health: UNKNOWN } } }
I20260806 09:22:13.786379 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.780800 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52398 (local address 127.16.86.129:45867)
0806 09:22:13.781096 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:13.781104 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:13.781127 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:22:13.781448 (+ 321us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:13.781453 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:13.781539 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:13.781710 (+ 171us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:22:13.782091 (+ 381us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.782831 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.784133 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.784551 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.784689 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.785147 (+ 458us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:22:13.785741 (+ 594us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:22:13.785899 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:13.785981 (+ 82us) server_negotiation.cc:299] Negotiation successful
0806 09:22:13.786070 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 09:22:13.792152 27025 raft_consensus.cc:1261] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa [term 1 FOLLOWER]: Refusing update from remote peer d3d9877aba6a445280ab89df34417458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:13.792587 27216 raft_consensus.cc:1261] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [term 1 FOLLOWER]: Refusing update from remote peer d3d9877aba6a445280ab89df34417458: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:13.793138 27448 consensus_queue.cc:1035] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cb7873ac0eb467ebce54384aee30caa" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33775 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:13.793718 27438 consensus_queue.cc:1035] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d8f51643bf134d19843fc65d7fd39502" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 36101 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:13.797398 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.796741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.130:58785)
0806 09:22:13.797034 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.797164 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20260806 09:22:13.797504 27435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.796602 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.131:34893)
0806 09:22:13.796938 (+ 336us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.797166 (+ 228us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260806 09:22:13.802567 27455 mvcc.cc:205] Tried to move back new op lower bound from 7315489316003819520 to 7315489315817852928. Current Snapshot: MvccSnapshot[applied={T|T < 7315489316003819520}]
I20260806 09:22:13.805320 27435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.802906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.131:42053)
0806 09:22:13.804908 (+ 2002us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.805061 (+ 153us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1829}
I20260806 09:22:13.808354 27454 mvcc.cc:205] Tried to move back new op lower bound from 7315489316003819520 to 7315489315817852928. Current Snapshot: MvccSnapshot[applied={T|T < 7315489316003819520 or (T in {7315489316003819520})}]
I20260806 09:22:13.809538 27442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.808927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.130:34257)
0806 09:22:13.809173 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.809299 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 09:22:13.816829 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.810725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.129:35481)
0806 09:22:13.816031 (+ 5306us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.816169 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5169}
I20260806 09:22:13.822959 27443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.819296 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.16.86.129:38893)
0806 09:22:13.822439 (+ 3143us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:13.822615 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":2950}
I20260806 09:22:13.831712 26620 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:13.832157 26620 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:51120:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:13.841123 26620 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dee9589594ac4444bff7783dc8b59f3b: Sending DeleteTablet for 3 replicas of tablet a937fa3212f949c2b488406d08d93a5d
I20260806 09:22:13.842351 26792 tablet_service.cc:1514] Processing DeleteTablet for tablet a937fa3212f949c2b488406d08d93a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:13 UTC) from {username='oryx', principal='oryx/127.16.86.188@KRBTEST.COM'} at 127.0.0.1:52388
I20260806 09:22:13.842825 27189 tablet_service.cc:1514] Processing DeleteTablet for tablet a937fa3212f949c2b488406d08d93a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:13 UTC) from {username='oryx', principal='oryx/127.16.86.188@KRBTEST.COM'} at 127.0.0.1:38960
I20260806 09:22:13.842953 26999 tablet_service.cc:1514] Processing DeleteTablet for tablet a937fa3212f949c2b488406d08d93a5d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:13 UTC) from {username='oryx', principal='oryx/127.16.86.188@KRBTEST.COM'} at 127.0.0.1:51470
I20260806 09:22:13.844981 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 26615
I20260806 09:22:13.858253 27465 tablet_replica.cc:330] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458: stopping tablet replica
I20260806 09:22:13.858795 27465 raft_consensus.cc:2229] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:22:13.859794 27465 raft_consensus.cc:2258] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:13.860011 27470 tablet_replica.cc:330] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa: stopping tablet replica
I20260806 09:22:13.860325 27470 raft_consensus.cc:2229] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:13.860771 27470 raft_consensus.cc:2258] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:13.860982 27465 ts_tablet_manager.cc:1905] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:13.864784 27470 ts_tablet_manager.cc:1905] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:13.868952 27468 tablet_replica.cc:330] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502: stopping tablet replica
I20260806 09:22:13.869293 27468 raft_consensus.cc:2229] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:13.869725 27468 raft_consensus.cc:2258] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:13.871593 27468 ts_tablet_manager.cc:1905] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:13.871968 27470 ts_tablet_manager.cc:1918] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:13.872160 27470 log.cc:1196] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/a937fa3212f949c2b488406d08d93a5d
I20260806 09:22:13.872643 27465 ts_tablet_manager.cc:1918] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:13.872725 27470 ts_tablet_manager.cc:1939] T a937fa3212f949c2b488406d08d93a5d P 5cb7873ac0eb467ebce54384aee30caa: Deleting consensus metadata
I20260806 09:22:13.872972 27465 log.cc:1196] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/a937fa3212f949c2b488406d08d93a5d
I20260806 09:22:13.873564 27465 ts_tablet_manager.cc:1939] T a937fa3212f949c2b488406d08d93a5d P d3d9877aba6a445280ab89df34417458: Deleting consensus metadata
I20260806 09:22:13.874385 26602 catalog_manager.cc:4841] TS 5cb7873ac0eb467ebce54384aee30caa (127.16.86.130:33775): tablet a937fa3212f949c2b488406d08d93a5d (table test-table [id=452c07ba43334704a47cdc36a3f25306]) successfully deleted
I20260806 09:22:13.875154 26601 catalog_manager.cc:4841] TS d3d9877aba6a445280ab89df34417458 (127.16.86.129:45867): tablet a937fa3212f949c2b488406d08d93a5d (table test-table [id=452c07ba43334704a47cdc36a3f25306]) successfully deleted
I20260806 09:22:13.879408 27468 ts_tablet_manager.cc:1918] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:13.879614 27468 log.cc:1196] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/a937fa3212f949c2b488406d08d93a5d
I20260806 09:22:13.880179 27468 ts_tablet_manager.cc:1939] T a937fa3212f949c2b488406d08d93a5d P d8f51643bf134d19843fc65d7fd39502: Deleting consensus metadata
I20260806 09:22:13.880702 27423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:13.847357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52408 (local address 127.16.86.129:45867)
0806 09:22:13.847647 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:13.847656 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:13.857269 (+ 9613us) server_negotiation.cc:407] Connection header received
0806 09:22:13.858463 (+ 1194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:13.858472 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:13.858699 (+ 227us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:13.858890 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:13.861621 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.862838 (+ 1217us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.864898 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:13.865648 (+ 750us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:13.865829 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:13.870369 (+ 4540us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:13.870395 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:13.870428 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:13.870494 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:13.876089 (+ 5595us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:13.877362 (+ 1273us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:13.877374 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:13.877379 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:13.877668 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:13.878234 (+ 566us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:13.878239 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:13.878243 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:13.878936 (+ 693us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:13.879324 (+ 388us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:13.879897 (+ 573us) server_negotiation.cc:299] Negotiation successful
0806 09:22:13.880201 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:22:13.881569 26603 catalog_manager.cc:4841] TS d8f51643bf134d19843fc65d7fd39502 (127.16.86.131:36101): tablet a937fa3212f949c2b488406d08d93a5d (table test-table [id=452c07ba43334704a47cdc36a3f25306]) successfully deleted
I20260806 09:22:13.882120 26876 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:14.000831 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 26615
I20260806 09:22:14.036264 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 26900
I20260806 09:22:14.057789 27426 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:14.037496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.16.86.130:33775)
0806 09:22:14.037858 (+ 362us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:14.037865 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:14.037893 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:22:14.038034 (+ 141us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:14.038038 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:14.038123 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:14.038238 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:14.040162 (+ 1924us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:14.041042 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:14.042694 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:14.043305 (+ 611us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:14.043440 (+ 135us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:14.047848 (+ 4408us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:14.047883 (+ 35us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:14.047914 (+ 31us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:14.048034 (+ 120us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:14.053040 (+ 5006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:14.054426 (+ 1386us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:14.054435 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:14.054437 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:14.054706 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:14.055622 (+ 916us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:14.055629 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:14.055632 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:14.056223 (+ 591us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:14.056654 (+ 431us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:14.056959 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 09:22:14.057262 (+ 303us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":188}
I20260806 09:22:14.058737 27073 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:14.075920 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:14.075367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:54358)
0806 09:22:14.075593 (+ 226us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:14.075712 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 09:22:14.077167 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:14.076781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:54362)
0806 09:22:14.076935 (+ 154us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:14.077033 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20260806 09:22:14.167745 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 26900
I20260806 09:22:14.199712 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 27099
I20260806 09:22:14.217192 27424 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:14.200719 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38966 (local address 127.16.86.131:36101)
0806 09:22:14.200970 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:14.200975 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:14.200998 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:22:14.201302 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:14.201308 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:14.201388 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:14.201489 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:14.203305 (+ 1816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:14.204263 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:14.205879 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:14.206329 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:14.206456 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:14.210018 (+ 3562us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:14.210038 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:14.210057 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:14.210103 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:14.213717 (+ 3614us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:14.214789 (+ 1072us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:14.214798 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:14.214800 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:14.214956 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:14.215494 (+ 538us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:14.215498 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:14.215501 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:14.215951 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:14.216237 (+ 286us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:14.216609 (+ 372us) server_negotiation.cc:299] Negotiation successful
0806 09:22:14.216839 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 09:22:14.218257 27277 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:14.366206 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 27099
I20260806 09:22:14.401065 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 26121
I20260806 09:22:14.402773 26369 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:14.502966 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:14.502329 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:54370)
0806 09:22:14.502577 (+ 248us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:14.502737 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20260806 09:22:14.504272 27425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:14.503864 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43733 (local address 127.0.0.1:54384)
0806 09:22:14.504038 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:14.504134 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:43733: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 09:22:14.530685 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 26121
I20260806 09:22:14.567260 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 26277
I20260806 09:22:14.568660 26668 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:14.704082 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 26277
2026-08-06T09:22:14Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (11655 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:22:14 dist-test-slave-h3ft krb5kdc[27484](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:22:14 dist-test-slave-h3ft krb5kdc[27484](info): set up 2 sockets
Aug 06 09:22:14 dist-test-slave-h3ft krb5kdc[27484](info): commencing operation
krb5kdc: starting...
W20260806 09:22:16.094928 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.326s user 0.005s sys 0.040s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:22:16 dist-test-slave-h3ft krb5kdc[27484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:22:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:22:16Z Disabled control of system clock
WARNING: no policy specified for oryx/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.190.keytab.
Entry for principal oryx/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.190.keytab.
I20260806 09:22:16.444334 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:35813
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.190.keytab
--principal=oryx/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:35813
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:16.885027 27499 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:16.885541 27499 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:16.885802 27499 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:16.894346 27499 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:22:16.894573 27499 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20260806 09:22:16.894696 27499 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:16.894840 27499 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:16.894948 27499 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:22:16.895113 27499 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:22:16.907480 27499 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:35813
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.190.keytab
--principal=oryx/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:35813
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:16.909405 27499 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:16.911221 27499 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:16.927170 27507 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:16.927290 27508 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:16.927229 27510 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:16.928522 27499 server_base.cc:1034] running on GCE node
I20260806 09:22:16.929719 27499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:16.931887 27499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:16.933225 27499 hybrid_clock.cc:648] HybridClock initialized: now 1786008136933129 us; error 54 us; skew 500 ppm
I20260806 09:22:16.933732 27499 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:16 dist-test-slave-h3ft krb5kdc[27484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008136, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:16.945108 27499 init.cc:377] Logged in from keytab as oryx/127.16.86.190@KRBTEST.COM (short username oryx)
I20260806 09:22:16.947616 27499 webserver.cc:460] Webserver started at http://127.16.86.190:46591/ using document root <none> and password file <none>
I20260806 09:22:16.948740 27499 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:16.948921 27499 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:16.949436 27499 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:16.953830 27499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "9ec383fe1ca2430fb73364f156a803fc"
format_stamp: "Formatted at 2026-08-06 09:22:16 on dist-test-slave-h3ft"
server_key: "2aa8e0cc1f1b520be4738cc0f329e2ea"
server_key_iv: "03ee1fe2ae7aef5b3e4ca8f74dfac345"
server_key_version: "encryptionkey@0"
I20260806 09:22:16.955125 27499 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "9ec383fe1ca2430fb73364f156a803fc"
format_stamp: "Formatted at 2026-08-06 09:22:16 on dist-test-slave-h3ft"
server_key: "2aa8e0cc1f1b520be4738cc0f329e2ea"
server_key_iv: "03ee1fe2ae7aef5b3e4ca8f74dfac345"
server_key_version: "encryptionkey@0"
I20260806 09:22:16.963178 27499 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260806 09:22:16.968289 27519 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:16.970429 27499 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:22:16.970715 27499 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "9ec383fe1ca2430fb73364f156a803fc"
format_stamp: "Formatted at 2026-08-06 09:22:16 on dist-test-slave-h3ft"
server_key: "2aa8e0cc1f1b520be4738cc0f329e2ea"
server_key_iv: "03ee1fe2ae7aef5b3e4ca8f74dfac345"
server_key_version: "encryptionkey@0"
I20260806 09:22:16.971105 27499 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:16.996548 27499 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:17.002686 27499 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:17.003075 27499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:17.031451 27499 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:35813
I20260806 09:22:17.031463 27607 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:35813 every 8 connection(s)
I20260806 09:22:17.033268 27499 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:22:17.038081 27609 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:17.041666 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 27499
I20260806 09:22:17.041980 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:22:17.042805 16730 external_mini_cluster.cc:1428] Setting key 0082cae635317821ce59a6ead903c8c0
I20260806 09:22:17.051218 27609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc: Bootstrap starting.
I20260806 09:22:17.055495 27609 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:17.056895 27609 log.cc:824] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:17.061623 27609 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc: No bootstrap required, opened a new log
Aug 06 09:22:17 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008136, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:17.072338 27609 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec383fe1ca2430fb73364f156a803fc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35813 } }
I20260806 09:22:17.072875 27609 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:17.073009 27609 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ec383fe1ca2430fb73364f156a803fc, State: Initialized, Role: FOLLOWER
I20260806 09:22:17.073698 27609 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec383fe1ca2430fb73364f156a803fc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35813 } }
I20260806 09:22:17.073987 27609 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:22:17.074170 27609 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:22:17.074390 27609 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:17.076925 27612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:17.046061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60172 (local address 127.16.86.190:35813)
0806 09:22:17.047433 (+ 1372us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:17.047450 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:17.047509 (+ 59us) server_negotiation.cc:407] Connection header received
0806 09:22:17.049925 (+ 2416us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:17.049958 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:17.050499 (+ 541us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:17.051051 (+ 552us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:17.053181 (+ 2130us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:17.054903 (+ 1722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:17.057073 (+ 2170us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:17.058468 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:17.058735 (+ 267us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:17.065405 (+ 6670us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:17.065424 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:17.065456 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:17.065549 (+ 93us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:17.070110 (+ 4561us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:17.071554 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:17.071563 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:17.071573 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:17.071841 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:17.072395 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:17.072401 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:17.072405 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:17.073498 (+ 1093us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:17.073897 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:17.074593 (+ 696us) server_negotiation.cc:299] Negotiation successful
0806 09:22:17.075280 (+ 687us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1054,"thread_start_us":672,"threads_started":1}
I20260806 09:22:17.077915 27609 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec383fe1ca2430fb73364f156a803fc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35813 } }
I20260806 09:22:17.078612 27609 leader_election.cc:302] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ec383fe1ca2430fb73364f156a803fc; no voters:
I20260806 09:22:17.079434 27609 leader_election.cc:288] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:22:17.079703 27616 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:17.082252 27609 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:22:17.083455 27616 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [term 1 LEADER]: Becoming Leader. State: Replica: 9ec383fe1ca2430fb73364f156a803fc, State: Running, Role: LEADER
I20260806 09:22:17.084368 27616 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec383fe1ca2430fb73364f156a803fc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35813 } }
I20260806 09:22:17.088240 27618 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ec383fe1ca2430fb73364f156a803fc. Latest consensus state: current_term: 1 leader_uuid: "9ec383fe1ca2430fb73364f156a803fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec383fe1ca2430fb73364f156a803fc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35813 } } }
I20260806 09:22:17.088382 27617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ec383fe1ca2430fb73364f156a803fc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ec383fe1ca2430fb73364f156a803fc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35813 } } }
I20260806 09:22:17.089025 27617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:17.089349 27618 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:17.090310 27629 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:22:17.097241 27629 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:22:17.109845 27629 catalog_manager.cc:1347] Generated new cluster ID: d4d4431bc3c94ebf894a2375bfa1026e
I20260806 09:22:17.110030 27629 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:22:17.128172 27629 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:22:17.130026 27629 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:22:17.144241 27629 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc: Generated new TSK 0
I20260806 09:22:17.145309 27629 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab.
Entry for principal oryx/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab.
I20260806 09:22:17.244632 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:35813
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab
--principal=oryx/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:17.630718 27644 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:17.631129 27644 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:17.631302 27644 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:17.638932 27644 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:17.639120 27644 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:17.639272 27644 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:22:17.639397 27644 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:22:17.650303 27644 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab
--principal=oryx/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.86.190:35813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:17.651971 27644 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:17.653486 27644 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:17.670459 27650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:17.670863 27652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:17.671065 27649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:17.672489 27644 server_base.cc:1034] running on GCE node
I20260806 09:22:17.673364 27644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:17.674645 27644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:17.675935 27644 hybrid_clock.cc:648] HybridClock initialized: now 1786008137675840 us; error 88 us; skew 500 ppm
I20260806 09:22:17.676370 27644 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:17 dist-test-slave-h3ft krb5kdc[27484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008137, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:17.680472 27644 init.cc:377] Logged in from keytab as oryx/127.16.86.129@KRBTEST.COM (short username oryx)
I20260806 09:22:17.682647 27644 webserver.cc:460] Webserver started at http://127.16.86.129:33441/ using document root <none> and password file <none>
I20260806 09:22:17.683686 27644 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:17.683837 27644 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:17.684233 27644 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:17.687995 27644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "8eb628a5f0774f82b90d04bd056b5099"
format_stamp: "Formatted at 2026-08-06 09:22:17 on dist-test-slave-h3ft"
server_key: "34ef83ceff05f61eb077dc38f7ebb28a"
server_key_iv: "22a0c738d673db431692fccab8fb2feb"
server_key_version: "encryptionkey@0"
I20260806 09:22:17.688936 27644 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "8eb628a5f0774f82b90d04bd056b5099"
format_stamp: "Formatted at 2026-08-06 09:22:17 on dist-test-slave-h3ft"
server_key: "34ef83ceff05f61eb077dc38f7ebb28a"
server_key_iv: "22a0c738d673db431692fccab8fb2feb"
server_key_version: "encryptionkey@0"
I20260806 09:22:17.695606 27644 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260806 09:22:17.700588 27664 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:17.702494 27644 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:22:17.702754 27644 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "8eb628a5f0774f82b90d04bd056b5099"
format_stamp: "Formatted at 2026-08-06 09:22:17 on dist-test-slave-h3ft"
server_key: "34ef83ceff05f61eb077dc38f7ebb28a"
server_key_iv: "22a0c738d673db431692fccab8fb2feb"
server_key_version: "encryptionkey@0"
I20260806 09:22:17.703101 27644 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:17.719355 27644 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:17.724356 27644 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:17.724692 27644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:17.728106 27644 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:17.728235 27644 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:17.728412 27644 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:17.728506 27644 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:22:17 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008137, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:17.762697 27676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:17.729502 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:33163 (local address 127.16.86.190:35813)
0806 09:22:17.730158 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:17.730166 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:17.731953 (+ 1787us) server_negotiation.cc:407] Connection header received
0806 09:22:17.736217 (+ 4264us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:17.736226 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:17.736366 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:17.736518 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:17.739600 (+ 3082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:17.740840 (+ 1240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:17.744693 (+ 3853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:17.745278 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:17.745482 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:17.752991 (+ 7509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:17.753001 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:17.753010 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:17.753082 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:17.757536 (+ 4454us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:17.758867 (+ 1331us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:17.758877 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:17.758884 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:17.759161 (+ 277us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:17.759991 (+ 830us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:17.760002 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:17.760007 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:17.760595 (+ 588us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:17.760838 (+ 243us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:17.761858 (+ 1020us) server_negotiation.cc:299] Negotiation successful
0806 09:22:17.762192 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":455,"thread_start_us":305,"threads_started":1}
I20260806 09:22:17.764742 27677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:17.730421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35813 (local address 127.16.86.129:33163)
0806 09:22:17.731655 (+ 1234us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:17.731740 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:17.735604 (+ 3864us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:17.736948 (+ 1344us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:17.736966 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:17.738052 (+ 1086us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:17.739238 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:17.739275 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:17.741197 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:17.741203 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:17.744446 (+ 3243us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:17.744475 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:17.745745 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:17.745758 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:17.746395 (+ 637us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:17.747641 (+ 1246us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:17.747689 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:17.752475 (+ 4786us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:17.757858 (+ 5383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:17.757870 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:17.757892 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:17.758645 (+ 753us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:17.759357 (+ 712us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:17.759365 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:17.759371 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:17.759773 (+ 402us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:17.760791 (+ 1018us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:17.760803 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:17.761351 (+ 548us) client_negotiation.cc:769] Sending connection context
0806 09:22:17.761718 (+ 367us) client_negotiation.cc:240] Negotiation successful
0806 09:22:17.762454 (+ 736us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":829,"thread_start_us":509,"threads_started":1}
I20260806 09:22:17.799024 27644 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:35331
I20260806 09:22:17.799084 27826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:35331 every 8 connection(s)
I20260806 09:22:17.800788 27644 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:22:17.804801 27827 heartbeater.cc:347] Connected to a master server at 127.16.86.190:35813
I20260806 09:22:17.805230 27783 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:17.776851 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35813 (local address 127.16.86.129:39283)
0806 09:22:17.783776 (+ 6925us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:17.783815 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:17.784070 (+ 255us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:17.784730 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:17.784742 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:17.785814 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:17.786567 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:17.786584 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:17.788164 (+ 1580us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:17.788173 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:17.790703 (+ 2530us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:17.790732 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:17.792566 (+ 1834us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:17.792576 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:17.792849 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:17.793900 (+ 1051us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:17.793930 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:17.795808 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:17.800743 (+ 4935us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:17.800750 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:17.800757 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:17.801505 (+ 748us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:17.802216 (+ 711us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:17.802223 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:17.802228 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:17.802456 (+ 228us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:17.803612 (+ 1156us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:17.803619 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:17.804017 (+ 398us) client_negotiation.cc:769] Sending connection context
0806 09:22:17.804181 (+ 164us) client_negotiation.cc:240] Negotiation successful
0806 09:22:17.804502 (+ 321us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6582,"thread_start_us":6269,"threads_started":1}
I20260806 09:22:17.805503 27676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:17.776881 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:39283 (local address 127.16.86.190:35813)
0806 09:22:17.777206 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:17.777219 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:17.784000 (+ 6781us) server_negotiation.cc:407] Connection header received
0806 09:22:17.784308 (+ 308us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:17.784317 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:17.784461 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:17.784663 (+ 202us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:17.786876 (+ 2213us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:17.787934 (+ 1058us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:17.791045 (+ 3111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:17.791938 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:17.792074 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:17.796183 (+ 4109us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:17.796191 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:17.796195 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:17.796251 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:17.800499 (+ 4248us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:17.801758 (+ 1259us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:17.801767 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:17.801773 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:17.802011 (+ 238us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:17.802658 (+ 647us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:17.802664 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:17.802667 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:17.803372 (+ 705us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:17.803532 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:17.804633 (+ 1101us) server_negotiation.cc:299] Negotiation successful
0806 09:22:17.805028 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 09:22:17.806030 27827 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:17.807368 27827 heartbeater.cc:511] Master 127.16.86.190:35813 requested a full tablet report, sending...
I20260806 09:22:17.807905 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 27644
I20260806 09:22:17.808169 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:22:17.809145 16730 external_mini_cluster.cc:1428] Setting key 1ec5a9e4d52fdc349a5df612ddc198a0
I20260806 09:22:17.812047 27541 ts_manager.cc:194] Registered new tserver with Master: 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331)
I20260806 09:22:17.814136 27541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:33163
WARNING: no policy specified for oryx/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab.
Entry for principal oryx/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab.
I20260806 09:22:17.877192 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:35813
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab
--principal=oryx/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:18.307309 27833 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:18.307698 27833 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:18.307876 27833 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:18.316581 27833 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:18.316736 27833 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:18.316834 27833 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:22:18.316927 27833 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:22:18.328452 27833 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab
--principal=oryx/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.86.190:35813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:18.330166 27833 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:18.332028 27833 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:18.348730 27840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:18.348711 27842 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:18.348644 27845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:18.349947 27833 server_base.cc:1034] running on GCE node
I20260806 09:22:18.350893 27833 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:18.352348 27833 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:18.353627 27833 hybrid_clock.cc:648] HybridClock initialized: now 1786008138353514 us; error 76 us; skew 500 ppm
I20260806 09:22:18.354094 27833 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:18 dist-test-slave-h3ft krb5kdc[27484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008138, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:18.358445 27833 init.cc:377] Logged in from keytab as oryx/127.16.86.130@KRBTEST.COM (short username oryx)
I20260806 09:22:18.360689 27833 webserver.cc:460] Webserver started at http://127.16.86.130:46141/ using document root <none> and password file <none>
I20260806 09:22:18.361714 27833 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:18.361822 27833 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:18.362278 27833 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:18.366078 27833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "773d982788d842f0904c34cf27d30512"
format_stamp: "Formatted at 2026-08-06 09:22:18 on dist-test-slave-h3ft"
server_key: "5d5c2ab417338f8a5109d00fe91ef023"
server_key_iv: "60ab8004131fde0e4e66644c210e27e7"
server_key_version: "encryptionkey@0"
I20260806 09:22:18.367203 27833 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "773d982788d842f0904c34cf27d30512"
format_stamp: "Formatted at 2026-08-06 09:22:18 on dist-test-slave-h3ft"
server_key: "5d5c2ab417338f8a5109d00fe91ef023"
server_key_iv: "60ab8004131fde0e4e66644c210e27e7"
server_key_version: "encryptionkey@0"
I20260806 09:22:18.374471 27833 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.001s sys 0.005s
I20260806 09:22:18.379465 27860 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:18.381290 27833 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:22:18.381541 27833 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "773d982788d842f0904c34cf27d30512"
format_stamp: "Formatted at 2026-08-06 09:22:18 on dist-test-slave-h3ft"
server_key: "5d5c2ab417338f8a5109d00fe91ef023"
server_key_iv: "60ab8004131fde0e4e66644c210e27e7"
server_key_version: "encryptionkey@0"
I20260806 09:22:18.381811 27833 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:18.408742 27833 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:18.414404 27833 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:18.414721 27833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:18.418319 27833 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:18.418460 27833 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:18.418630 27833 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:18.418716 27833 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:22:18 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008138, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:18.446857 27873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:18.419121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:54251 (local address 127.16.86.190:35813)
0806 09:22:18.419956 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:18.419960 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:18.421700 (+ 1740us) server_negotiation.cc:407] Connection header received
0806 09:22:18.424639 (+ 2939us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:18.424644 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:18.424756 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:18.424889 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:18.427765 (+ 2876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:18.428958 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:18.432072 (+ 3114us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:18.432610 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:18.432756 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:18.438997 (+ 6241us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:18.439057 (+ 60us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:18.439065 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:18.439136 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:18.443310 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:18.444229 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:18.444233 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:18.444236 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:18.444372 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:18.444864 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:18.444868 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:18.444870 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:18.445160 (+ 290us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:18.445354 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:18.446146 (+ 792us) server_negotiation.cc:299] Negotiation successful
0806 09:22:18.446416 (+ 270us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":704,"thread_start_us":590,"threads_started":1}
I20260806 09:22:18.448239 27876 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:18.420190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35813 (local address 127.16.86.130:54251)
0806 09:22:18.421444 (+ 1254us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:18.421510 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:18.424257 (+ 2747us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:18.425240 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:18.425256 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:18.426327 (+ 1071us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:18.427535 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:18.427565 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:18.429137 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:18.429141 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:18.431807 (+ 2666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:18.431836 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:18.432941 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:18.432949 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:18.433195 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:18.434471 (+ 1276us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:18.434497 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:18.438585 (+ 4088us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:18.443502 (+ 4917us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:18.443514 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:18.443547 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:18.444064 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:18.444494 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:18.444499 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:18.444503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:18.444727 (+ 224us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:18.445309 (+ 582us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:18.445319 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:18.445753 (+ 434us) client_negotiation.cc:769] Sending connection context
0806 09:22:18.446042 (+ 289us) client_negotiation.cc:240] Negotiation successful
0806 09:22:18.446621 (+ 579us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":834,"thread_start_us":544,"threads_started":1}
I20260806 09:22:18.476271 27873 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:18.457828 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:41365 (local address 127.16.86.190:35813)
0806 09:22:18.458051 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:18.458058 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:18.459302 (+ 1244us) server_negotiation.cc:407] Connection header received
0806 09:22:18.459546 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:18.459551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:18.459668 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:18.459800 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:18.460937 (+ 1137us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:18.461698 (+ 761us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:18.463121 (+ 1423us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:18.463709 (+ 588us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:18.463864 (+ 155us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:18.467106 (+ 3242us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:18.467115 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:18.467122 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:18.467177 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:18.472159 (+ 4982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:18.473498 (+ 1339us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:18.473507 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:18.473513 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:18.473728 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:18.474373 (+ 645us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:18.474381 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:18.474385 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:18.474814 (+ 429us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:18.475068 (+ 254us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:18.475581 (+ 513us) server_negotiation.cc:299] Negotiation successful
0806 09:22:18.475889 (+ 308us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 09:22:18.476553 27961 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:18.458162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35813 (local address 127.16.86.130:41365)
0806 09:22:18.459154 (+ 992us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:18.459181 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:18.459386 (+ 205us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:18.459842 (+ 456us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:18.459847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:18.460451 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:18.460775 (+ 324us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:18.460784 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:18.461848 (+ 1064us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:18.461856 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:18.462919 (+ 1063us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:18.462932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:18.463969 (+ 1037us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:18.463977 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:18.464325 (+ 348us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:18.465418 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:18.465439 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:18.466817 (+ 1378us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:18.472418 (+ 5601us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:18.472428 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:18.472435 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:18.473199 (+ 764us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:18.473886 (+ 687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:18.473892 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:18.473897 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:18.474163 (+ 266us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:18.474963 (+ 800us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:18.474974 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:18.475270 (+ 296us) client_negotiation.cc:769] Sending connection context
0806 09:22:18.475457 (+ 187us) client_negotiation.cc:240] Negotiation successful
0806 09:22:18.476055 (+ 598us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":708,"thread_start_us":440,"threads_started":1}
I20260806 09:22:18.486950 27833 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:46593
I20260806 09:22:18.486979 28024 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:46593 every 8 connection(s)
I20260806 09:22:18.488724 27833 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:22:18.491444 28025 heartbeater.cc:347] Connected to a master server at 127.16.86.190:35813
I20260806 09:22:18.491756 28025 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:18.492496 28025 heartbeater.cc:511] Master 127.16.86.190:35813 requested a full tablet report, sending...
I20260806 09:22:18.494226 27541 ts_manager.cc:194] Registered new tserver with Master: 773d982788d842f0904c34cf27d30512 (127.16.86.130:46593)
I20260806 09:22:18.494477 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 27833
I20260806 09:22:18.494666 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:22:18.495563 16730 external_mini_cluster.cc:1428] Setting key 7776009e3d19a5a07b23fa25c334da09
I20260806 09:22:18.495817 27541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:54251
WARNING: no policy specified for oryx/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab.
Entry for principal oryx/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab.
I20260806 09:22:18.571805 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:35813
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab
--principal=oryx/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:40375
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:22:18.818444 27827 heartbeater.cc:503] Master 127.16.86.190:35813 was elected leader, sending a full tablet report...
W20260806 09:22:19.025959 28032 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:19.026455 28032 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:19.026651 28032 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:19.035668 28032 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:19.035933 28032 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:19.036106 28032 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20260806 09:22:19.036259 28032 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:22:19.048128 28032 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:40375
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab
--principal=oryx/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--enable_txn_system_client_init=true
--tserver_master_addrs=127.16.86.190:35813
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:19.050034 28032 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:19.051981 28032 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:19.070720 28039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:19.071293 28040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:19.071312 28043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:19.073129 28032 server_base.cc:1034] running on GCE node
I20260806 09:22:19.074385 28032 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:19.079562 28032 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:19.083232 28032 hybrid_clock.cc:648] HybridClock initialized: now 1786008139082887 us; error 241 us; skew 500 ppm
I20260806 09:22:19.083935 28032 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:19 dist-test-slave-h3ft krb5kdc[27484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008139, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:19.088904 28032 init.cc:377] Logged in from keytab as oryx/127.16.86.131@KRBTEST.COM (short username oryx)
I20260806 09:22:19.091784 28032 webserver.cc:460] Webserver started at http://127.16.86.131:41403/ using document root <none> and password file <none>
I20260806 09:22:19.092931 28032 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:19.093125 28032 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:19.093629 28032 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:19.097821 28032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "823c87d9387b401f8142d88e1a746930"
format_stamp: "Formatted at 2026-08-06 09:22:19 on dist-test-slave-h3ft"
server_key: "5661c3ee3fe97e445ec06e44800888ed"
server_key_iv: "b531158c95977834508775979858289f"
server_key_version: "encryptionkey@0"
I20260806 09:22:19.099192 28032 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "823c87d9387b401f8142d88e1a746930"
format_stamp: "Formatted at 2026-08-06 09:22:19 on dist-test-slave-h3ft"
server_key: "5661c3ee3fe97e445ec06e44800888ed"
server_key_iv: "b531158c95977834508775979858289f"
server_key_version: "encryptionkey@0"
I20260806 09:22:19.108080 28032 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260806 09:22:19.114419 28053 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:19.116822 28032 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 09:22:19.117167 28032 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "823c87d9387b401f8142d88e1a746930"
format_stamp: "Formatted at 2026-08-06 09:22:19 on dist-test-slave-h3ft"
server_key: "5661c3ee3fe97e445ec06e44800888ed"
server_key_iv: "b531158c95977834508775979858289f"
server_key_version: "encryptionkey@0"
I20260806 09:22:19.117548 28032 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:19.140977 28032 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:19.147989 28032 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:19.148443 28032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:19.153126 28032 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:19.153306 28032 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:19.153503 28032 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:19.153642 28032 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:22:19 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008139, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:19.187887 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:19.153920 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:53955 (local address 127.16.86.190:35813)
0806 09:22:19.155226 (+ 1306us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:19.155231 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:19.156975 (+ 1744us) server_negotiation.cc:407] Connection header received
0806 09:22:19.160088 (+ 3113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:19.160096 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:19.160248 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:19.160482 (+ 234us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:19.163814 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:19.165273 (+ 1459us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:19.168605 (+ 3332us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:19.169532 (+ 927us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:19.169717 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:19.177543 (+ 7826us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:19.177549 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:19.177552 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:19.177594 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:19.181703 (+ 4109us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:19.183049 (+ 1346us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:19.183058 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:19.183064 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:19.183252 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:19.183969 (+ 717us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:19.183975 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:19.183978 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:19.184444 (+ 466us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:19.184628 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:19.186921 (+ 2293us) server_negotiation.cc:299] Negotiation successful
0806 09:22:19.187296 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1151,"thread_start_us":978,"threads_started":1}
I20260806 09:22:19.191730 28070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:19.155433 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35813 (local address 127.16.86.131:53955)
0806 09:22:19.156717 (+ 1284us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:19.156790 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:19.159547 (+ 2757us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:19.160962 (+ 1415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:19.160980 (+ 18us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:19.162329 (+ 1349us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:19.163444 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:19.163474 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:19.165640 (+ 2166us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:19.165645 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:19.168303 (+ 2658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:19.168329 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:19.169881 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:19.169887 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:19.170129 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:19.171238 (+ 1109us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:19.171292 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:19.177096 (+ 5804us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:19.181929 (+ 4833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:19.181940 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:19.181967 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:19.182835 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:19.183419 (+ 584us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:19.183423 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:19.183425 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:19.183771 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:19.185238 (+ 1467us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:19.185255 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:19.186192 (+ 937us) client_negotiation.cc:769] Sending connection context
0806 09:22:19.188335 (+ 2143us) client_negotiation.cc:240] Negotiation successful
0806 09:22:19.189226 (+ 891us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":871,"thread_start_us":549,"threads_started":1}
I20260806 09:22:19.213972 28032 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:37941
I20260806 09:22:19.214011 28225 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:37941 every 8 connection(s)
I20260806 09:22:19.215922 28032 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:22:19.219367 28226 heartbeater.cc:347] Connected to a master server at 127.16.86.190:35813
I20260806 09:22:19.219745 28226 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:19.220886 28226 heartbeater.cc:511] Master 127.16.86.190:35813 requested a full tablet report, sending...
I20260806 09:22:19.222748 28202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:19.201570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35813 (local address 127.16.86.131:44527)
0806 09:22:19.203636 (+ 2066us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:19.203669 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:19.203885 (+ 216us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:19.204523 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:19.204529 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:19.205228 (+ 699us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:19.205643 (+ 415us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:19.205656 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:19.207540 (+ 1884us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:19.207546 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:19.209160 (+ 1614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:19.209181 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:19.210525 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:19.210539 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:19.210977 (+ 438us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:19.212072 (+ 1095us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:19.212104 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:19.213792 (+ 1688us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:19.218994 (+ 5202us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:19.219027 (+ 33us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:19.219035 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:19.219825 (+ 790us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:19.220357 (+ 532us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:19.220362 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:19.220365 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:19.220564 (+ 199us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:19.221376 (+ 812us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:19.221382 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:19.221619 (+ 237us) client_negotiation.cc:769] Sending connection context
0806 09:22:19.221800 (+ 181us) client_negotiation.cc:240] Negotiation successful
0806 09:22:19.222221 (+ 421us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1869,"thread_start_us":1680,"threads_started":1}
I20260806 09:22:19.222934 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:19.201677 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:44527 (local address 127.16.86.190:35813)
0806 09:22:19.202012 (+ 335us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:19.202024 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:19.203825 (+ 1801us) server_negotiation.cc:407] Connection header received
0806 09:22:19.204062 (+ 237us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:19.204071 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:19.204189 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:19.204355 (+ 166us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:19.205876 (+ 1521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:19.207299 (+ 1423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:19.209360 (+ 2061us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:19.210127 (+ 767us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:19.210308 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:19.214108 (+ 3800us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:19.214117 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:19.214125 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:19.214175 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:19.218776 (+ 4601us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:19.220027 (+ 1251us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:19.220032 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:19.220035 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:19.220212 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:19.220766 (+ 554us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:19.220771 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:19.220773 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:19.221221 (+ 448us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:19.221373 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:19.222041 (+ 668us) server_negotiation.cc:299] Negotiation successful
0806 09:22:19.222388 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 09:22:19.222923 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 28032
I20260806 09:22:19.223183 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:22:19.223177 27541 ts_manager.cc:194] Registered new tserver with Master: 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941)
I20260806 09:22:19.223852 16730 external_mini_cluster.cc:1428] Setting key 7c4be9c415c3546e74ea446eaa22a2c7
I20260806 09:22:19.224496 27541 master_service.cc:496] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:53955
I20260806 09:22:19.228883 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
Aug 06 09:22:19 dist-test-slave-h3ft krb5kdc[27484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Aug 06 09:22:19 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:19 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:19.500108 28025 heartbeater.cc:503] Master 127.16.86.190:35813 was elected leader, sending a full tablet report...
I20260806 09:22:20.228395 28226 heartbeater.cc:503] Master 127.16.86.190:35813 was elected leader, sending a full tablet report...
W20260806 09:22:20.458247 28239 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.155s user 0.000s sys 0.027s
W20260806 09:22:20.459828 28068 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:19.295662 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34844 (local address 127.16.86.190:35813)
0806 09:22:19.295970 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:19.295979 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:19.296365 (+ 386us) server_negotiation.cc:407] Connection header received
0806 09:22:19.296623 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:19.296627 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:19.296708 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:19.296846 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:19.298663 (+ 1817us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:19.299527 (+ 864us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:19.301026 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:19.301445 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:19.301555 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.459520 (+1157965us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34844: BlockingRecv error: recv got EOF from 127.0.0.1:34844 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 06 09:22:20 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008139, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:20.481865 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.464769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34852 (local address 127.16.86.190:35813)
0806 09:22:20.465180 (+ 411us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:20.465191 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:20.465497 (+ 306us) server_negotiation.cc:407] Connection header received
0806 09:22:20.465782 (+ 285us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:20.465787 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:20.465862 (+ 75us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:20.465987 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:20.467676 (+ 1689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.468502 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.469890 (+ 1388us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.470399 (+ 509us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.470551 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.475308 (+ 4757us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:20.475315 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:20.475318 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:20.475356 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:20.478751 (+ 3395us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:20.479583 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:20.479591 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:20.479594 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:20.479755 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:20.480207 (+ 452us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:20.480212 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:20.480215 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:20.480544 (+ 329us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:20.480854 (+ 310us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:20.481232 (+ 378us) server_negotiation.cc:299] Negotiation successful
0806 09:22:20.481498 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":211}
I20260806 09:22:20.488520 27540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:22:20.498085 27540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:22:20.533981 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.516370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40686 (local address 127.16.86.131:37941)
0806 09:22:20.523256 (+ 6886us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:20.523268 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:20.523307 (+ 39us) server_negotiation.cc:407] Connection header received
0806 09:22:20.523806 (+ 499us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:20.523818 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:20.524192 (+ 374us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:20.524500 (+ 308us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:20.526095 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.527917 (+ 1822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.530768 (+ 2851us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.532405 (+ 1637us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.532566 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.532857 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:20.533398 (+ 541us) server_negotiation.cc:299] Negotiation successful
0806 09:22:20.533592 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6682,"spinlock_wait_cycles":32640,"thread_start_us":6658,"threads_started":1}
I20260806 09:22:20.534144 28258 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.515705 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:37941 (local address 127.0.0.1:40686)
0806 09:22:20.522915 (+ 7210us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:20.522952 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:20.523505 (+ 553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:20.524905 (+ 1400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:20.524913 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:20.524946 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:20.525819 (+ 873us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:20.525834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.528115 (+ 2281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.528123 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:20.530594 (+ 2471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:20.530608 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.532980 (+ 2372us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.532985 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:20.533128 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.533165 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 09:22:20.533339 (+ 174us) client_negotiation.cc:240] Negotiation successful
0806 09:22:20.533576 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6996,"mutex_wait_us":34,"thread_start_us":6473,"threads_started":1}
I20260806 09:22:20.535455 28262 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.523223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:46593 (local address 127.0.0.1:34080)
0806 09:22:20.523911 (+ 688us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:20.523937 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:20.524046 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:20.525370 (+ 1324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:20.525376 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:20.525405 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:20.526385 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:20.526401 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.528357 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.528364 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:20.531506 (+ 3142us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:20.531520 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.533900 (+ 2380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.533906 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:20.534316 (+ 410us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.534497 (+ 181us) client_negotiation.cc:769] Sending connection context
0806 09:22:20.534594 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 09:22:20.534908 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"mutex_wait_us":567,"thread_start_us":403,"threads_started":1}
I20260806 09:22:20.535506 28257 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.515472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35331 (local address 127.0.0.1:43010)
0806 09:22:20.516619 (+ 1147us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:20.516717 (+ 98us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:20.516998 (+ 281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:20.525208 (+ 8210us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:20.525215 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:20.525253 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:20.526310 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:20.526329 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.528147 (+ 1818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.528153 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:20.531502 (+ 3349us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:20.531520 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.533899 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.533906 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:20.534314 (+ 408us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.534495 (+ 181us) client_negotiation.cc:769] Sending connection context
0806 09:22:20.534594 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 09:22:20.534858 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":785,"mutex_wait_us":398,"thread_start_us":872,"threads_started":1}
I20260806 09:22:20.536451 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.516107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43010 (local address 127.16.86.129:35331)
0806 09:22:20.524454 (+ 8347us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:20.524466 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:20.524504 (+ 38us) server_negotiation.cc:407] Connection header received
0806 09:22:20.524636 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:20.524645 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:20.524946 (+ 301us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:20.525208 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:20.526565 (+ 1357us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.527983 (+ 1418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.531739 (+ 3756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.533546 (+ 1807us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.535621 (+ 2075us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.535852 (+ 231us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:20.535928 (+ 76us) server_negotiation.cc:299] Negotiation successful
0806 09:22:20.536128 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8129,"thread_start_us":8394,"threads_started":1}
I20260806 09:22:20.535692 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.523535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.16.86.130:46593)
0806 09:22:20.524696 (+ 1161us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:20.524707 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:20.524736 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:22:20.524862 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:20.524870 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:20.525109 (+ 239us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:20.525551 (+ 442us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:20.526652 (+ 1101us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.528187 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.531774 (+ 3587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.533547 (+ 1773us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.534723 (+ 1176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.534978 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:20.535113 (+ 135us) server_negotiation.cc:299] Negotiation successful
0806 09:22:20.535311 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":967,"thread_start_us":733,"threads_started":1}
I20260806 09:22:20.538858 28125 tablet_service.cc:1467] Processing CreateTablet for tablet b8896192ac7d48f28b7891ba00259e14 (DEFAULT_TABLE table=test-table [id=cb75309b57d24ab1be1fd8ad7d82ba20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:20.540768 28125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8896192ac7d48f28b7891ba00259e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:20.542177 27730 tablet_service.cc:1467] Processing CreateTablet for tablet b8896192ac7d48f28b7891ba00259e14 (DEFAULT_TABLE table=test-table [id=cb75309b57d24ab1be1fd8ad7d82ba20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:20.543938 27730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8896192ac7d48f28b7891ba00259e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:20.544085 27931 tablet_service.cc:1467] Processing CreateTablet for tablet b8896192ac7d48f28b7891ba00259e14 (DEFAULT_TABLE table=test-table [id=cb75309b57d24ab1be1fd8ad7d82ba20]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:20.545807 27931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet b8896192ac7d48f28b7891ba00259e14. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:20.560144 28264 tablet_bootstrap.cc:492] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930: Bootstrap starting.
I20260806 09:22:20.562939 28268 tablet_bootstrap.cc:492] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512: Bootstrap starting.
I20260806 09:22:20.565573 28264 tablet_bootstrap.cc:654] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:20.566388 28266 tablet_bootstrap.cc:492] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099: Bootstrap starting.
I20260806 09:22:20.567955 28268 tablet_bootstrap.cc:654] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:20.569133 28264 log.cc:824] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:20.569541 28268 log.cc:824] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:20.570544 28266 tablet_bootstrap.cc:654] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:20.571889 28266 log.cc:824] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:20.574261 28268 tablet_bootstrap.cc:492] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512: No bootstrap required, opened a new log
I20260806 09:22:20.574509 28264 tablet_bootstrap.cc:492] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930: No bootstrap required, opened a new log
I20260806 09:22:20.574576 28268 ts_tablet_manager.cc:1397] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20260806 09:22:20.574877 28264 ts_tablet_manager.cc:1397] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930: Time spent bootstrapping tablet: real 0.015s user 0.009s sys 0.002s
I20260806 09:22:20.575335 28266 tablet_bootstrap.cc:492] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099: No bootstrap required, opened a new log
I20260806 09:22:20.575601 28266 ts_tablet_manager.cc:1397] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20260806 09:22:20.585868 28268 raft_consensus.cc:348] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } }
I20260806 09:22:20.586469 28268 raft_consensus.cc:374] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:20.586154 28264 raft_consensus.cc:348] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } }
I20260806 09:22:20.586639 28268 raft_consensus.cc:729] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773d982788d842f0904c34cf27d30512, State: Initialized, Role: FOLLOWER
I20260806 09:22:20.586481 28266 raft_consensus.cc:348] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } }
I20260806 09:22:20.586766 28264 raft_consensus.cc:374] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:20.586920 28264 raft_consensus.cc:729] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823c87d9387b401f8142d88e1a746930, State: Initialized, Role: FOLLOWER
I20260806 09:22:20.586954 28266 raft_consensus.cc:374] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:20.587293 28266 raft_consensus.cc:729] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb628a5f0774f82b90d04bd056b5099, State: Initialized, Role: FOLLOWER
I20260806 09:22:20.587567 28268 consensus_queue.cc:260] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } }
I20260806 09:22:20.587745 28264 consensus_queue.cc:260] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } }
I20260806 09:22:20.588027 28266 consensus_queue.cc:260] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } }
I20260806 09:22:20.591871 28268 ts_tablet_manager.cc:1428] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512: Time spent starting tablet: real 0.017s user 0.014s sys 0.001s
I20260806 09:22:20.593494 28266 ts_tablet_manager.cc:1428] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099: Time spent starting tablet: real 0.018s user 0.005s sys 0.011s
I20260806 09:22:20.594807 28273 raft_consensus.cc:482] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:22:20.595438 28273 raft_consensus.cc:504] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } }
I20260806 09:22:20.596168 28264 ts_tablet_manager.cc:1428] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930: Time spent starting tablet: real 0.021s user 0.014s sys 0.003s
I20260806 09:22:20.603304 28273 leader_election.cc:288] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331), 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941)
I20260806 09:22:20.609822 28277 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.599375 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:37941 (local address 127.16.86.130:43071)
0806 09:22:20.601783 (+ 2408us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:20.601816 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:20.602490 (+ 674us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:20.602983 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:20.602991 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:20.603058 (+ 67us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:20.603788 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:20.603803 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.605315 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.605320 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:20.607297 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:20.607311 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.608990 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.608997 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:20.609180 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.609235 (+ 55us) client_negotiation.cc:769] Sending connection context
0806 09:22:20.609346 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 09:22:20.609500 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2165,"thread_start_us":3507,"threads_started":1}
I20260806 09:22:20.610049 28261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.600280 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:43071 (local address 127.16.86.131:37941)
0806 09:22:20.600504 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:20.600511 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:20.602275 (+ 1764us) server_negotiation.cc:407] Connection header received
0806 09:22:20.602737 (+ 462us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:20.602741 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:20.602807 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:20.602937 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:20.603999 (+ 1062us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.605163 (+ 1164us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.607472 (+ 2309us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.608751 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.608912 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.608983 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:20.609493 (+ 510us) server_negotiation.cc:299] Negotiation successful
0806 09:22:20.609682 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:22:20.610594 28259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.600170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:48635 (local address 127.16.86.129:35331)
0806 09:22:20.600511 (+ 341us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:20.600517 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:20.603488 (+ 2971us) server_negotiation.cc:407] Connection header received
0806 09:22:20.603746 (+ 258us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:20.603752 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:20.603828 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:20.603927 (+ 99us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:20.604738 (+ 811us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.605605 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.608208 (+ 2603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.609268 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.609425 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.609496 (+ 71us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:20.610129 (+ 633us) server_negotiation.cc:299] Negotiation successful
0806 09:22:20.610346 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":203}
I20260806 09:22:20.611466 28276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.599522 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35331 (local address 127.16.86.130:48635)
0806 09:22:20.603346 (+ 3824us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:20.603374 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:20.603592 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:20.603982 (+ 390us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:20.603987 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:20.604015 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:20.604495 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:20.604509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.605774 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.605779 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:20.607972 (+ 2193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:20.607991 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.609588 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.609596 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:20.609791 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.609889 (+ 98us) client_negotiation.cc:769] Sending connection context
0806 09:22:20.610827 (+ 938us) client_negotiation.cc:240] Negotiation successful
0806 09:22:20.611063 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3588,"thread_start_us":2385,"threads_started":1}
I20260806 09:22:20.611408 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8896192ac7d48f28b7891ba00259e14" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823c87d9387b401f8142d88e1a746930" is_pre_election: true
I20260806 09:22:20.612082 28154 raft_consensus.cc:2454] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 0.
I20260806 09:22:20.612798 27862 leader_election.cc:302] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773d982788d842f0904c34cf27d30512, 823c87d9387b401f8142d88e1a746930; no voters:
I20260806 09:22:20.612705 27765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8896192ac7d48f28b7891ba00259e14" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb628a5f0774f82b90d04bd056b5099" is_pre_election: true
I20260806 09:22:20.613210 28273 raft_consensus.cc:2781] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:22:20.613343 28273 raft_consensus.cc:482] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:22:20.613473 28273 raft_consensus.cc:3037] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:20.613530 27765 raft_consensus.cc:2454] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 0.
I20260806 09:22:20.616084 28273 raft_consensus.cc:504] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } }
I20260806 09:22:20.616806 28273 leader_election.cc:288] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331), 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941)
I20260806 09:22:20.617465 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8896192ac7d48f28b7891ba00259e14" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823c87d9387b401f8142d88e1a746930"
I20260806 09:22:20.617445 27765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "b8896192ac7d48f28b7891ba00259e14" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb628a5f0774f82b90d04bd056b5099"
I20260806 09:22:20.617638 28154 raft_consensus.cc:3037] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:20.617744 27765 raft_consensus.cc:3037] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:20.620283 28154 raft_consensus.cc:2454] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 1.
I20260806 09:22:20.621028 27862 leader_election.cc:302] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773d982788d842f0904c34cf27d30512, 823c87d9387b401f8142d88e1a746930; no voters:
I20260806 09:22:20.621317 28273 raft_consensus.cc:2781] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:20.621359 27765 raft_consensus.cc:2454] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 1.
I20260806 09:22:20.621888 28273 raft_consensus.cc:686] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 1 LEADER]: Becoming Leader. State: Replica: 773d982788d842f0904c34cf27d30512, State: Running, Role: LEADER
I20260806 09:22:20.622643 28273 consensus_queue.cc:237] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } }
I20260806 09:22:20.626824 27541 catalog_manager.cc:5495] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 reported cstate change: term changed from 0 to 1, leader changed from <none> to 773d982788d842f0904c34cf27d30512 (127.16.86.130). New cstate: current_term: 1 leader_uuid: "773d982788d842f0904c34cf27d30512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } health_report { overall_health: HEALTHY } } }
I20260806 09:22:20.662295 28263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.655848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34090 (local address 127.16.86.130:46593)
0806 09:22:20.656160 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:20.656168 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:20.656325 (+ 157us) server_negotiation.cc:407] Connection header received
0806 09:22:20.656677 (+ 352us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:20.656682 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:20.656780 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:20.656973 (+ 193us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:22:20.657387 (+ 414us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.658204 (+ 817us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.659709 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.660192 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.660335 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.660834 (+ 499us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:22:20.661621 (+ 787us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:22:20.661798 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:20.661854 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 09:22:20.661964 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 09:22:20.673319 28154 raft_consensus.cc:1261] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Refusing update from remote peer 773d982788d842f0904c34cf27d30512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:20.673319 27765 raft_consensus.cc:1261] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Refusing update from remote peer 773d982788d842f0904c34cf27d30512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:20.674665 28278 consensus_queue.cc:1035] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:20.675099 28273 consensus_queue.cc:1035] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:20.704711 27541 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:20.705060 27541 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:20.709664 27541 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc: Sending DeleteTablet for 3 replicas of tablet b8896192ac7d48f28b7891ba00259e14
I20260806 09:22:20.711313 27730 tablet_service.cc:1514] Processing DeleteTablet for tablet b8896192ac7d48f28b7891ba00259e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:20 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:43010
I20260806 09:22:20.711604 28125 tablet_service.cc:1514] Processing DeleteTablet for tablet b8896192ac7d48f28b7891ba00259e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:20 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:40686
I20260806 09:22:20.711665 27931 tablet_service.cc:1514] Processing DeleteTablet for tablet b8896192ac7d48f28b7891ba00259e14 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:20 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:34080
I20260806 09:22:20.714178 27541 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34852:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:22:20.716208 27541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:22:20.731362 27930 tablet_service.cc:1467] Processing CreateTablet for tablet ba72dc97571740d28994dc2f0fc071e9 (DEFAULT_TABLE table=test-table [id=bbe4fd7ccedf4d519d71ae85531686c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:20.731613 28121 tablet_service.cc:1467] Processing CreateTablet for tablet ba72dc97571740d28994dc2f0fc071e9 (DEFAULT_TABLE table=test-table [id=bbe4fd7ccedf4d519d71ae85531686c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:20.732296 27930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba72dc97571740d28994dc2f0fc071e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:20.732404 28121 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba72dc97571740d28994dc2f0fc071e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:20.735798 27729 tablet_service.cc:1467] Processing CreateTablet for tablet ba72dc97571740d28994dc2f0fc071e9 (DEFAULT_TABLE table=test-table [id=bbe4fd7ccedf4d519d71ae85531686c1]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:20.736570 27729 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ba72dc97571740d28994dc2f0fc071e9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:20.740538 28264 tablet_bootstrap.cc:492] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930: Bootstrap starting.
I20260806 09:22:20.743739 28264 tablet_bootstrap.cc:654] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:20.759508 28268 tablet_bootstrap.cc:492] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512: Bootstrap starting.
I20260806 09:22:20.762032 28268 tablet_bootstrap.cc:654] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:20.766182 28268 tablet_bootstrap.cc:492] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512: No bootstrap required, opened a new log
I20260806 09:22:20.766433 28268 ts_tablet_manager.cc:1397] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20260806 09:22:20.767509 28268 raft_consensus.cc:348] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.767795 28266 tablet_bootstrap.cc:492] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099: Bootstrap starting.
I20260806 09:22:20.767875 28268 raft_consensus.cc:374] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:20.767970 28268 raft_consensus.cc:729] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773d982788d842f0904c34cf27d30512, State: Initialized, Role: FOLLOWER
I20260806 09:22:20.768271 28268 consensus_queue.cc:260] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.769034 28268 ts_tablet_manager.cc:1428] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 09:22:20.769912 28266 tablet_bootstrap.cc:654] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:20.788877 28266 tablet_bootstrap.cc:492] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099: No bootstrap required, opened a new log
I20260806 09:22:20.789224 28266 ts_tablet_manager.cc:1397] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099: Time spent bootstrapping tablet: real 0.022s user 0.005s sys 0.005s
I20260806 09:22:20.790486 28266 raft_consensus.cc:348] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.790917 28266 raft_consensus.cc:374] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:20.791137 28266 raft_consensus.cc:729] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb628a5f0774f82b90d04bd056b5099, State: Initialized, Role: FOLLOWER
I20260806 09:22:20.791676 28266 consensus_queue.cc:260] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.792099 28302 tablet_replica.cc:330] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512: stopping tablet replica
I20260806 09:22:20.795800 28302 raft_consensus.cc:2229] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:22:20.797621 28266 ts_tablet_manager.cc:1428] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20260806 09:22:20.799233 28302 raft_consensus.cc:2258] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:20.799677 28300 tablet_replica.cc:330] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099: stopping tablet replica
I20260806 09:22:20.801221 28264 tablet_bootstrap.cc:492] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930: No bootstrap required, opened a new log
I20260806 09:22:20.801522 28264 ts_tablet_manager.cc:1397] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930: Time spent bootstrapping tablet: real 0.061s user 0.019s sys 0.030s
I20260806 09:22:20.802335 28300 raft_consensus.cc:2229] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:20.802856 28264 raft_consensus.cc:348] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.803073 28300 raft_consensus.cc:2258] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:20.803212 28264 raft_consensus.cc:374] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:20.803359 28264 raft_consensus.cc:729] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823c87d9387b401f8142d88e1a746930, State: Initialized, Role: FOLLOWER
I20260806 09:22:20.803723 28264 consensus_queue.cc:260] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.804572 28301 tablet_replica.cc:330] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930: stopping tablet replica
I20260806 09:22:20.805071 28301 raft_consensus.cc:2229] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 09:22:20.805884 27828 tablet.cc:2369] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:22:20.806011 28264 ts_tablet_manager.cc:1428] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260806 09:22:20.807224 28301 raft_consensus.cc:2258] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:20.809753 28278 raft_consensus.cc:482] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:22:20.810027 28278 raft_consensus.cc:504] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.810875 28278 leader_election.cc:288] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331), 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941)
I20260806 09:22:20.811718 27765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba72dc97571740d28994dc2f0fc071e9" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb628a5f0774f82b90d04bd056b5099" is_pre_election: true
I20260806 09:22:20.811929 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba72dc97571740d28994dc2f0fc071e9" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823c87d9387b401f8142d88e1a746930" is_pre_election: true
I20260806 09:22:20.812016 27765 raft_consensus.cc:2454] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 0.
I20260806 09:22:20.812198 28154 raft_consensus.cc:2454] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 0.
I20260806 09:22:20.812713 27863 leader_election.cc:302] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773d982788d842f0904c34cf27d30512, 8eb628a5f0774f82b90d04bd056b5099; no voters:
I20260806 09:22:20.813321 28278 raft_consensus.cc:2781] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:22:20.813447 28278 raft_consensus.cc:482] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:22:20.813570 28278 raft_consensus.cc:3037] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:20.817453 28278 raft_consensus.cc:504] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.819697 28302 ts_tablet_manager.cc:1905] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:20.819650 27765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba72dc97571740d28994dc2f0fc071e9" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb628a5f0774f82b90d04bd056b5099"
I20260806 09:22:20.819957 27765 raft_consensus.cc:3037] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:20.822496 28278 leader_election.cc:288] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331), 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941)
I20260806 09:22:20.823086 28301 ts_tablet_manager.cc:1905] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:20.823494 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ba72dc97571740d28994dc2f0fc071e9" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823c87d9387b401f8142d88e1a746930"
I20260806 09:22:20.824498 28154 raft_consensus.cc:3037] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:20.824828 28300 ts_tablet_manager.cc:1905] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:20.828477 28154 raft_consensus.cc:2454] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 1.
I20260806 09:22:20.829262 27862 leader_election.cc:302] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773d982788d842f0904c34cf27d30512, 823c87d9387b401f8142d88e1a746930; no voters:
I20260806 09:22:20.829712 28278 raft_consensus.cc:2781] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:20.829947 28278 raft_consensus.cc:686] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 1 LEADER]: Becoming Leader. State: Replica: 773d982788d842f0904c34cf27d30512, State: Running, Role: LEADER
I20260806 09:22:20.830304 28278 consensus_queue.cc:237] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.831974 28302 ts_tablet_manager.cc:1918] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:20.832221 28302 log.cc:1196] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/b8896192ac7d48f28b7891ba00259e14
I20260806 09:22:20.832257 27765 raft_consensus.cc:2454] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 1.
I20260806 09:22:20.832882 28302 ts_tablet_manager.cc:1939] T b8896192ac7d48f28b7891ba00259e14 P 773d982788d842f0904c34cf27d30512: Deleting consensus metadata
I20260806 09:22:20.835062 27520 catalog_manager.cc:4841] TS 773d982788d842f0904c34cf27d30512 (127.16.86.130:46593): tablet b8896192ac7d48f28b7891ba00259e14 (table test-table [id=cb75309b57d24ab1be1fd8ad7d82ba20]) successfully deleted
I20260806 09:22:20.840413 28301 ts_tablet_manager.cc:1918] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:20.840704 28301 log.cc:1196] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/b8896192ac7d48f28b7891ba00259e14
I20260806 09:22:20.841306 28301 ts_tablet_manager.cc:1939] T b8896192ac7d48f28b7891ba00259e14 P 823c87d9387b401f8142d88e1a746930: Deleting consensus metadata
I20260806 09:22:20.843660 27520 catalog_manager.cc:4841] TS 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941): tablet b8896192ac7d48f28b7891ba00259e14 (table test-table [id=cb75309b57d24ab1be1fd8ad7d82ba20]) successfully deleted
I20260806 09:22:20.843761 28300 ts_tablet_manager.cc:1918] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:20.844297 28300 log.cc:1196] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/b8896192ac7d48f28b7891ba00259e14
I20260806 09:22:20.845378 28300 ts_tablet_manager.cc:1939] T b8896192ac7d48f28b7891ba00259e14 P 8eb628a5f0774f82b90d04bd056b5099: Deleting consensus metadata
I20260806 09:22:20.844981 27541 catalog_manager.cc:5495] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 reported cstate change: term changed from 0 to 1, leader changed from <none> to 773d982788d842f0904c34cf27d30512 (127.16.86.130). New cstate: current_term: 1 leader_uuid: "773d982788d842f0904c34cf27d30512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } health_report { overall_health: UNKNOWN } } }
I20260806 09:22:20.849510 27521 catalog_manager.cc:4841] TS 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331): tablet b8896192ac7d48f28b7891ba00259e14 (table test-table [id=cb75309b57d24ab1be1fd8ad7d82ba20]) successfully deleted
Aug 06 09:22:20 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008136, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.190@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:20.914386 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.894053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35813 (local address 127.0.0.1:34856)
0806 09:22:20.894989 (+ 936us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:20.895055 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:20.895354 (+ 299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:20.895859 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:20.895900 (+ 41us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:20.897201 (+ 1301us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:20.897621 (+ 420us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:20.897635 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.898943 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.898949 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:20.900739 (+ 1790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:20.900756 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.901781 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.901788 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:20.901987 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.902930 (+ 943us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:20.902957 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:20.906596 (+ 3639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:20.911271 (+ 4675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:20.911289 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:20.911297 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:20.911935 (+ 638us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:20.912422 (+ 487us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:20.912425 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:20.912427 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:20.912604 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:20.913382 (+ 778us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:20.913389 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:20.913637 (+ 248us) client_negotiation.cc:769] Sending connection context
0806 09:22:20.913800 (+ 163us) client_negotiation.cc:240] Negotiation successful
0806 09:22:20.914038 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":462,"threads_started":1}
I20260806 09:22:20.914781 28068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:20.894007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34856 (local address 127.16.86.190:35813)
0806 09:22:20.894377 (+ 370us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:20.894393 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:20.895225 (+ 832us) server_negotiation.cc:407] Connection header received
0806 09:22:20.895575 (+ 350us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:20.895580 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:20.895661 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:20.895769 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:20.897829 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.898775 (+ 946us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.900952 (+ 2177us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:20.901463 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:20.901699 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:20.906937 (+ 5238us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:20.906945 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:20.906951 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:20.907052 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:20.910989 (+ 3937us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:20.912111 (+ 1122us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:20.912117 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:20.912122 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:20.912296 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:20.912743 (+ 447us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:20.912749 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:20.912753 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:20.913177 (+ 424us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:20.913349 (+ 172us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:20.913958 (+ 609us) server_negotiation.cc:299] Negotiation successful
0806 09:22:20.914237 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":175}
I20260806 09:22:20.921568 27541 catalog_manager.cc:1908] Servicing CreateTable request from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:34856:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20260806 09:22:20.923076 27541 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:22:20.931661 27730 tablet_service.cc:1467] Processing CreateTablet for tablet 38eceedc98f7490b86e7f1dded9e8010 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=40551d994c5543138ef90815004a843c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:22:20.932102 28125 tablet_service.cc:1467] Processing CreateTablet for tablet 38eceedc98f7490b86e7f1dded9e8010 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=40551d994c5543138ef90815004a843c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:22:20.932369 27730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eceedc98f7490b86e7f1dded9e8010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:20.932271 27931 tablet_service.cc:1467] Processing CreateTablet for tablet 38eceedc98f7490b86e7f1dded9e8010 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=40551d994c5543138ef90815004a843c]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20260806 09:22:20.932709 28125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eceedc98f7490b86e7f1dded9e8010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:20.933076 27931 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 38eceedc98f7490b86e7f1dded9e8010. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:20.940577 28264 tablet_bootstrap.cc:492] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930: Bootstrap starting.
I20260806 09:22:20.940848 28266 tablet_bootstrap.cc:492] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099: Bootstrap starting.
I20260806 09:22:20.941996 28268 tablet_bootstrap.cc:492] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: Bootstrap starting.
I20260806 09:22:20.942660 28266 tablet_bootstrap.cc:654] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:20.942670 28264 tablet_bootstrap.cc:654] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:20.944159 28268 tablet_bootstrap.cc:654] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:20.953806 28264 tablet_bootstrap.cc:492] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930: No bootstrap required, opened a new log
I20260806 09:22:20.954036 28264 ts_tablet_manager.cc:1397] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.007s
I20260806 09:22:20.954720 28266 tablet_bootstrap.cc:492] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099: No bootstrap required, opened a new log
I20260806 09:22:20.954959 28266 ts_tablet_manager.cc:1397] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.006s
I20260806 09:22:20.954895 28264 raft_consensus.cc:348] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.955574 28264 raft_consensus.cc:374] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:20.955683 28264 raft_consensus.cc:729] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823c87d9387b401f8142d88e1a746930, State: Initialized, Role: FOLLOWER
I20260806 09:22:20.956022 28264 consensus_queue.cc:260] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.956152 28266 raft_consensus.cc:348] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.956631 28266 raft_consensus.cc:374] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:20.956770 28266 raft_consensus.cc:729] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb628a5f0774f82b90d04bd056b5099, State: Initialized, Role: FOLLOWER
I20260806 09:22:20.956527 28274 tablet_replica.cc:439] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } }
I20260806 09:22:20.956931 28274 tablet_replica.cc:442] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:22:20.957235 28266 consensus_queue.cc:260] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.957717 28268 tablet_bootstrap.cc:492] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: No bootstrap required, opened a new log
I20260806 09:22:20.957973 28268 ts_tablet_manager.cc:1397] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.009s
I20260806 09:22:20.957783 28275 tablet_replica.cc:439] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } }
I20260806 09:22:20.958165 28275 tablet_replica.cc:442] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:22:20.958978 28268 raft_consensus.cc:348] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.959340 28268 raft_consensus.cc:374] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:20.959438 28268 raft_consensus.cc:729] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773d982788d842f0904c34cf27d30512, State: Initialized, Role: FOLLOWER
I20260806 09:22:20.959784 28268 consensus_queue.cc:260] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:20.960402 28278 tablet_replica.cc:439] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } }
I20260806 09:22:20.960732 28278 tablet_replica.cc:442] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:22:20.960739 28266 ts_tablet_manager.cc:1428] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260806 09:22:20.962088 28268 ts_tablet_manager.cc:1428] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260806 09:22:20.963719 28264 ts_tablet_manager.cc:1428] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
W20260806 09:22:20.972591 28228 tablet.cc:2369] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:22:20.994468 28028 tablet.cc:2369] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:22:21.161839 28274 raft_consensus.cc:482] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:22:21.162071 28274 raft_consensus.cc:504] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.163990 28274 leader_election.cc:288] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331), 773d982788d842f0904c34cf27d30512 (127.16.86.130:46593)
I20260806 09:22:21.173923 28278 raft_consensus.cc:482] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:22:21.174278 28278 raft_consensus.cc:504] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.175413 28278 leader_election.cc:288] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331), 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941)
I20260806 09:22:21.177302 27765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eceedc98f7490b86e7f1dded9e8010" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb628a5f0774f82b90d04bd056b5099" is_pre_election: true
I20260806 09:22:21.177723 27765 raft_consensus.cc:2454] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 0.
I20260806 09:22:21.178246 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eceedc98f7490b86e7f1dded9e8010" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823c87d9387b401f8142d88e1a746930" is_pre_election: true
I20260806 09:22:21.178638 28154 raft_consensus.cc:2454] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 0.
I20260806 09:22:21.179349 27862 leader_election.cc:302] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773d982788d842f0904c34cf27d30512, 823c87d9387b401f8142d88e1a746930; no voters:
I20260806 09:22:21.180227 28278 raft_consensus.cc:2781] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:22:21.180397 28278 raft_consensus.cc:482] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:22:21.180526 28278 raft_consensus.cc:3037] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:21.182857 28322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.164472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:40853 (local address 127.16.86.129:35331)
0806 09:22:21.174794 (+ 10322us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:21.174800 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:21.174827 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:22:21.174911 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:21.174915 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:21.174989 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:21.175137 (+ 148us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:21.175815 (+ 678us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.177098 (+ 1283us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.179851 (+ 2753us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.181118 (+ 1267us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.181286 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.181362 (+ 76us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:21.182364 (+ 1002us) server_negotiation.cc:299] Negotiation successful
0806 09:22:21.182514 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10001,"thread_start_us":10167,"threads_started":1}
I20260806 09:22:21.183223 28321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.164344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35331 (local address 127.16.86.131:40853)
0806 09:22:21.174549 (+ 10205us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:21.174578 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:21.174777 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:21.175157 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:21.175162 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:21.175187 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:21.175577 (+ 390us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:21.175593 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.177255 (+ 1662us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.177259 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:21.179652 (+ 2393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:21.179670 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.181459 (+ 1789us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.181465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:21.181631 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.181663 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:22:21.181756 (+ 93us) client_negotiation.cc:240] Negotiation successful
0806 09:22:21.181917 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10040,"thread_start_us":10518,"threads_started":1}
I20260806 09:22:21.184247 27765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eceedc98f7490b86e7f1dded9e8010" candidate_uuid: "823c87d9387b401f8142d88e1a746930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb628a5f0774f82b90d04bd056b5099" is_pre_election: true
I20260806 09:22:21.184569 27765 raft_consensus.cc:2454] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 823c87d9387b401f8142d88e1a746930 in term 0.
I20260806 09:22:21.185140 28056 leader_election.cc:302] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 823c87d9387b401f8142d88e1a746930, 8eb628a5f0774f82b90d04bd056b5099; no voters:
I20260806 09:22:21.185050 28278 raft_consensus.cc:504] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.185685 28274 raft_consensus.cc:2781] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:22:21.185809 28274 raft_consensus.cc:482] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:22:21.185920 28274 raft_consensus.cc:3037] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:21.186699 27765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eceedc98f7490b86e7f1dded9e8010" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb628a5f0774f82b90d04bd056b5099"
I20260806 09:22:21.186978 27765 raft_consensus.cc:3037] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:21.187219 28154 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eceedc98f7490b86e7f1dded9e8010" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823c87d9387b401f8142d88e1a746930"
I20260806 09:22:21.187734 28278 leader_election.cc:288] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331), 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941)
I20260806 09:22:21.189427 28274 raft_consensus.cc:504] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.190204 28274 leader_election.cc:288] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331), 773d982788d842f0904c34cf27d30512 (127.16.86.130:46593)
I20260806 09:22:21.190382 27765 raft_consensus.cc:2454] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 1.
I20260806 09:22:21.190421 28154 raft_consensus.cc:2379] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 773d982788d842f0904c34cf27d30512 in current term 1: Already voted for candidate 823c87d9387b401f8142d88e1a746930 in this term.
I20260806 09:22:21.190883 27863 leader_election.cc:302] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773d982788d842f0904c34cf27d30512, 8eb628a5f0774f82b90d04bd056b5099; no voters:
I20260806 09:22:21.191409 27763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eceedc98f7490b86e7f1dded9e8010" candidate_uuid: "823c87d9387b401f8142d88e1a746930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb628a5f0774f82b90d04bd056b5099"
I20260806 09:22:21.191725 27763 raft_consensus.cc:2379] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 823c87d9387b401f8142d88e1a746930 in current term 1: Already voted for candidate 773d982788d842f0904c34cf27d30512 in this term.
I20260806 09:22:21.192631 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.165372 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:43723 (local address 127.16.86.130:46593)
0806 09:22:21.176665 (+ 11293us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:21.176672 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:21.176690 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:22:21.176760 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:21.176765 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:21.176857 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:21.176948 (+ 91us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:21.178097 (+ 1149us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.179461 (+ 1364us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.182096 (+ 2635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.184951 (+ 2855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.185169 (+ 218us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.185253 (+ 84us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:21.191702 (+ 6449us) server_negotiation.cc:299] Negotiation successful
0806 09:22:21.192233 (+ 531us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10553,"thread_start_us":17644,"threads_started":1}
I20260806 09:22:21.193028 28278 raft_consensus.cc:2781] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:21.193106 28323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.164742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:46593 (local address 127.16.86.131:43723)
0806 09:22:21.176357 (+ 11615us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:21.176377 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:21.176493 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:21.177229 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:21.177237 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:21.177264 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:21.177824 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:21.177905 (+ 81us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.179720 (+ 1815us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.179728 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:21.181910 (+ 2182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:21.181923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.185561 (+ 3638us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.185569 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:21.191306 (+ 5737us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.191345 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 09:22:21.191456 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 09:22:21.191910 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11491,"thread_start_us":23119,"threads_started":1}
I20260806 09:22:21.193567 28278 raft_consensus.cc:686] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [term 1 LEADER]: Becoming Leader. State: Replica: 773d982788d842f0904c34cf27d30512, State: Running, Role: LEADER
I20260806 09:22:21.194409 28278 consensus_queue.cc:237] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.194546 27966 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eceedc98f7490b86e7f1dded9e8010" candidate_uuid: "823c87d9387b401f8142d88e1a746930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773d982788d842f0904c34cf27d30512" is_pre_election: true
I20260806 09:22:21.195581 27964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "38eceedc98f7490b86e7f1dded9e8010" candidate_uuid: "823c87d9387b401f8142d88e1a746930" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773d982788d842f0904c34cf27d30512"
I20260806 09:22:21.197094 28055 leader_election.cc:302] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 823c87d9387b401f8142d88e1a746930; no voters: 773d982788d842f0904c34cf27d30512, 8eb628a5f0774f82b90d04bd056b5099
I20260806 09:22:21.197463 28274 raft_consensus.cc:2726] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260806 09:22:21.202067 28292 tablet_replica.cc:439] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: TxnStatusTablet state changed. Reason: New leader 773d982788d842f0904c34cf27d30512. Latest consensus state: current_term: 1 leader_uuid: "773d982788d842f0904c34cf27d30512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } }
I20260806 09:22:21.202490 28292 tablet_replica.cc:442] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:22:21.203401 28326 txn_status_manager.cc:873] Waiting until node catch up with all replicated operations in previous term...
I20260806 09:22:21.203703 28326 txn_status_manager.cc:929] Loading transaction status metadata into memory...
I20260806 09:22:21.204613 27540 catalog_manager.cc:5495] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 reported cstate change: term changed from 0 to 1, leader changed from <none> to 773d982788d842f0904c34cf27d30512 (127.16.86.130). New cstate: current_term: 1 leader_uuid: "773d982788d842f0904c34cf27d30512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } health_report { overall_health: UNKNOWN } } }
I20260806 09:22:21.280246 28313 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.272503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:46593 (local address 127.0.0.1:34098)
0806 09:22:21.272932 (+ 429us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:21.272951 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:21.273155 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:21.274019 (+ 864us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:21.274025 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:21.274043 (+ 18us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:22:21.274307 (+ 264us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:21.274338 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.276204 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.276213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:21.277576 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:21.277591 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.278765 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.278770 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:21.278932 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.279051 (+ 119us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:22:21.279597 (+ 546us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:22:21.279666 (+ 69us) client_negotiation.cc:769] Sending connection context
0806 09:22:21.279765 (+ 99us) client_negotiation.cc:240] Negotiation successful
0806 09:22:21.279919 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":185}
I20260806 09:22:21.280251 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.272631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34098 (local address 127.16.86.130:46593)
0806 09:22:21.272812 (+ 181us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:21.272818 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:21.273047 (+ 229us) server_negotiation.cc:407] Connection header received
0806 09:22:21.273505 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:21.273513 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:21.273646 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:21.273773 (+ 127us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:22:21.274540 (+ 767us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.276016 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.277756 (+ 1740us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.278562 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.278720 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.279285 (+ 565us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:22:21.279473 (+ 188us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:22:21.279583 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:21.279794 (+ 211us) server_negotiation.cc:299] Negotiation successful
0806 09:22:21.279903 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20260806 09:22:21.286433 27765 raft_consensus.cc:1261] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Refusing update from remote peer 773d982788d842f0904c34cf27d30512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:21.286581 28154 raft_consensus.cc:1261] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Refusing update from remote peer 773d982788d842f0904c34cf27d30512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:21.287202 28278 consensus_queue.cc:1035] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:21.287752 28292 consensus_queue.cc:1035] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:21.296517 28275 tablet_replica.cc:439] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099: TxnStatusTablet state changed. Reason: New leader 773d982788d842f0904c34cf27d30512. Latest consensus state: current_term: 1 leader_uuid: "773d982788d842f0904c34cf27d30512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } }
I20260806 09:22:21.296883 28275 tablet_replica.cc:442] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:22:21.298621 28328 mvcc.cc:205] Tried to move back new op lower bound from 7315489346701701120 to 7315489346337157120. Current Snapshot: MvccSnapshot[applied={T|T < 7315489346701701120}]
I20260806 09:22:21.301759 28275 tablet_replica.cc:439] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "773d982788d842f0904c34cf27d30512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } }
I20260806 09:22:21.302119 28275 tablet_replica.cc:442] T 38eceedc98f7490b86e7f1dded9e8010 P 8eb628a5f0774f82b90d04bd056b5099: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:22:21.304371 28274 tablet_replica.cc:439] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930: TxnStatusTablet state changed. Reason: New leader 773d982788d842f0904c34cf27d30512. Latest consensus state: current_term: 1 leader_uuid: "773d982788d842f0904c34cf27d30512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } }
I20260806 09:22:21.304721 28274 tablet_replica.cc:442] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:22:21.306285 28292 tablet_replica.cc:439] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "773d982788d842f0904c34cf27d30512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } }
I20260806 09:22:21.306973 28292 tablet_replica.cc:442] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:22:21.315248 28292 tablet_replica.cc:439] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "773d982788d842f0904c34cf27d30512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } }
I20260806 09:22:21.315932 28292 tablet_replica.cc:442] T 38eceedc98f7490b86e7f1dded9e8010 P 773d982788d842f0904c34cf27d30512: This TxnStatusTablet replica's current role is: LEADER
I20260806 09:22:21.319850 28274 tablet_replica.cc:439] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "773d982788d842f0904c34cf27d30512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } } }
I20260806 09:22:21.320266 28274 tablet_replica.cc:442] T 38eceedc98f7490b86e7f1dded9e8010 P 823c87d9387b401f8142d88e1a746930: This TxnStatusTablet replica's current role is: FOLLOWER
I20260806 09:22:21.336865 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.329357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:45291 (local address 127.16.86.130:46593)
0806 09:22:21.329618 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:21.329631 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:21.330272 (+ 641us) server_negotiation.cc:407] Connection header received
0806 09:22:21.330447 (+ 175us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:21.330460 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:21.330587 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:21.330739 (+ 152us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:22:21.331393 (+ 654us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.332921 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.334555 (+ 1634us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.335365 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.335534 (+ 169us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.336022 (+ 488us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:22:21.336127 (+ 105us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:22:21.336233 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:21.336378 (+ 145us) server_negotiation.cc:299] Negotiation successful
0806 09:22:21.336462 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:22:21.336999 28342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.329360 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:46593 (local address 127.16.86.130:45291)
0806 09:22:21.330146 (+ 786us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:21.330164 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:21.330301 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:21.330786 (+ 485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:21.330789 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:21.330799 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
0806 09:22:21.331198 (+ 399us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:21.331209 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.333123 (+ 1914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.333127 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:21.334368 (+ 1241us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:21.334380 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.335591 (+ 1211us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.335595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:21.335739 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.335807 (+ 68us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
0806 09:22:21.336243 (+ 436us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
0806 09:22:21.336255 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 09:22:21.336349 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 09:22:21.336490 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"mutex_wait_us":1,"thread_start_us":415,"threads_started":1}
I20260806 09:22:21.346990 28154 raft_consensus.cc:1261] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Refusing update from remote peer 773d982788d842f0904c34cf27d30512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:21.347199 27765 raft_consensus.cc:1261] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Refusing update from remote peer 773d982788d842f0904c34cf27d30512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:21.347697 28278 consensus_queue.cc:1035] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:21.348052 28292 consensus_queue.cc:1035] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:21.424659 27538 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:21.425107 27538 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:34852:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:21.429447 27538 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc: Sending DeleteTablet for 3 replicas of tablet ba72dc97571740d28994dc2f0fc071e9
I20260806 09:22:21.430516 27730 tablet_service.cc:1514] Processing DeleteTablet for tablet ba72dc97571740d28994dc2f0fc071e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:21 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:43010
I20260806 09:22:21.431409 28125 tablet_service.cc:1514] Processing DeleteTablet for tablet ba72dc97571740d28994dc2f0fc071e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:21 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:40686
I20260806 09:22:21.432142 27930 tablet_service.cc:1514] Processing DeleteTablet for tablet ba72dc97571740d28994dc2f0fc071e9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:21 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:34080
I20260806 09:22:21.445817 28350 tablet_replica.cc:330] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099: stopping tablet replica
I20260806 09:22:21.446126 28350 raft_consensus.cc:2229] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:21.446475 28350 raft_consensus.cc:2258] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:21.446765 28353 tablet_replica.cc:330] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512: stopping tablet replica
I20260806 09:22:21.447113 28353 raft_consensus.cc:2229] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:22:21.447674 28353 raft_consensus.cc:2258] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:21.448017 28350 ts_tablet_manager.cc:1905] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:21.449455 28353 ts_tablet_manager.cc:1905] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:21.456317 28350 ts_tablet_manager.cc:1918] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:22:21.456496 28350 log.cc:1196] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/ba72dc97571740d28994dc2f0fc071e9
I20260806 09:22:21.457018 28350 ts_tablet_manager.cc:1939] T ba72dc97571740d28994dc2f0fc071e9 P 8eb628a5f0774f82b90d04bd056b5099: Deleting consensus metadata
I20260806 09:22:21.458382 27521 catalog_manager.cc:4841] TS 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331): tablet ba72dc97571740d28994dc2f0fc071e9 (table test-table [id=bbe4fd7ccedf4d519d71ae85531686c1]) successfully deleted
I20260806 09:22:21.460913 28351 tablet_replica.cc:330] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930: stopping tablet replica
I20260806 09:22:21.461781 28351 raft_consensus.cc:2229] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:21.462212 28351 raft_consensus.cc:2258] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:21.463739 28353 ts_tablet_manager.cc:1918] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:22:21.463817 28351 ts_tablet_manager.cc:1905] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:21.464059 28353 log.cc:1196] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/ba72dc97571740d28994dc2f0fc071e9
I20260806 09:22:21.464778 28353 ts_tablet_manager.cc:1939] T ba72dc97571740d28994dc2f0fc071e9 P 773d982788d842f0904c34cf27d30512: Deleting consensus metadata
I20260806 09:22:21.466356 27520 catalog_manager.cc:4841] TS 773d982788d842f0904c34cf27d30512 (127.16.86.130:46593): tablet ba72dc97571740d28994dc2f0fc071e9 (table test-table [id=bbe4fd7ccedf4d519d71ae85531686c1]) successfully deleted
I20260806 09:22:21.474048 28351 ts_tablet_manager.cc:1918] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:22:21.474254 28351 log.cc:1196] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/ba72dc97571740d28994dc2f0fc071e9
I20260806 09:22:21.474841 28351 ts_tablet_manager.cc:1939] T ba72dc97571740d28994dc2f0fc071e9 P 823c87d9387b401f8142d88e1a746930: Deleting consensus metadata
I20260806 09:22:21.476596 27520 catalog_manager.cc:4841] TS 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941): tablet ba72dc97571740d28994dc2f0fc071e9 (table test-table [id=bbe4fd7ccedf4d519d71ae85531686c1]) successfully deleted
Aug 06 09:22:21 dist-test-slave-h3ft krb5kdc[27484](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Aug 06 09:22:21 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:21 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 09:22:21.540714 28363 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:21.513053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34872 (local address 127.16.86.190:35813)
0806 09:22:21.524156 (+ 11103us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:21.524165 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:21.524201 (+ 36us) server_negotiation.cc:407] Connection header received
0806 09:22:21.524326 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:21.524334 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:21.524449 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:21.524637 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:21.526736 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.528204 (+ 1468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.530387 (+ 2183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.531284 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.531462 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.540357 (+ 8895us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34872: BlockingRecv error: recv got EOF from 127.0.0.1:34872 (error 108)
Metrics: {"server-negotiator.queue_time_us":10908,"thread_start_us":10632,"threads_started":1}
Aug 06 09:22:21 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM
I20260806 09:22:21.563366 28363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.544355 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34882 (local address 127.16.86.190:35813)
0806 09:22:21.544559 (+ 204us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:21.544568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:21.544996 (+ 428us) server_negotiation.cc:407] Connection header received
0806 09:22:21.545225 (+ 229us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:21.545230 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:21.545307 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:21.545419 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:21.546777 (+ 1358us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.547644 (+ 867us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.548907 (+ 1263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.549338 (+ 431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.549482 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.555085 (+ 5603us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:21.555096 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:21.555103 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:21.555177 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:21.560012 (+ 4835us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:21.560854 (+ 842us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:21.560861 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:21.560866 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:21.561077 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:21.561500 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:21.561506 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:21.561510 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:21.561962 (+ 452us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:21.562273 (+ 311us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:21.562566 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 09:22:21.562862 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 09:22:21.568238 27538 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:22:21.569052 27538 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:22:21.577574 27730 tablet_service.cc:1467] Processing CreateTablet for tablet 2b41a672b9dd4631ac442a3d06a03475 (DEFAULT_TABLE table=test-table [id=d78bcb9c20d14fbbb36768677ef2310e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:21.578121 27730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b41a672b9dd4631ac442a3d06a03475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:21.578148 28125 tablet_service.cc:1467] Processing CreateTablet for tablet 2b41a672b9dd4631ac442a3d06a03475 (DEFAULT_TABLE table=test-table [id=d78bcb9c20d14fbbb36768677ef2310e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:21.578118 27930 tablet_service.cc:1467] Processing CreateTablet for tablet 2b41a672b9dd4631ac442a3d06a03475 (DEFAULT_TABLE table=test-table [id=d78bcb9c20d14fbbb36768677ef2310e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:21.578615 28125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b41a672b9dd4631ac442a3d06a03475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:21.578753 27930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 2b41a672b9dd4631ac442a3d06a03475. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:21.589283 28373 tablet_bootstrap.cc:492] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930: Bootstrap starting.
I20260806 09:22:21.590159 28372 tablet_bootstrap.cc:492] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099: Bootstrap starting.
I20260806 09:22:21.591055 28374 tablet_bootstrap.cc:492] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512: Bootstrap starting.
I20260806 09:22:21.591655 28373 tablet_bootstrap.cc:654] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:21.592501 28372 tablet_bootstrap.cc:654] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:21.595093 28374 tablet_bootstrap.cc:654] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:21.595510 28373 tablet_bootstrap.cc:492] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930: No bootstrap required, opened a new log
I20260806 09:22:21.595759 28373 ts_tablet_manager.cc:1397] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20260806 09:22:21.596815 28373 raft_consensus.cc:348] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.597136 28373 raft_consensus.cc:374] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:21.597254 28373 raft_consensus.cc:729] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823c87d9387b401f8142d88e1a746930, State: Initialized, Role: FOLLOWER
I20260806 09:22:21.597671 28373 consensus_queue.cc:260] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.602079 28373 ts_tablet_manager.cc:1428] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260806 09:22:21.603163 28372 tablet_bootstrap.cc:492] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099: No bootstrap required, opened a new log
I20260806 09:22:21.603613 28372 ts_tablet_manager.cc:1397] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260806 09:22:21.604856 28372 raft_consensus.cc:348] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.605239 28372 raft_consensus.cc:374] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:21.605340 28372 raft_consensus.cc:729] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb628a5f0774f82b90d04bd056b5099, State: Initialized, Role: FOLLOWER
I20260806 09:22:21.605711 28372 consensus_queue.cc:260] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.607486 28372 ts_tablet_manager.cc:1428] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260806 09:22:21.609525 28374 tablet_bootstrap.cc:492] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512: No bootstrap required, opened a new log
I20260806 09:22:21.609707 28374 ts_tablet_manager.cc:1397] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.005s
I20260806 09:22:21.610602 28374 raft_consensus.cc:348] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.610802 28374 raft_consensus.cc:374] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:21.610880 28374 raft_consensus.cc:729] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773d982788d842f0904c34cf27d30512, State: Initialized, Role: FOLLOWER
I20260806 09:22:21.611231 28374 consensus_queue.cc:260] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.611974 28374 ts_tablet_manager.cc:1428] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260806 09:22:21.683131 28275 raft_consensus.cc:482] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:22:21.683502 28275 raft_consensus.cc:504] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.685964 28275 leader_election.cc:288] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 773d982788d842f0904c34cf27d30512 (127.16.86.130:46593), 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941)
I20260806 09:22:21.699959 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.686396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:39803 (local address 127.16.86.130:46593)
0806 09:22:21.686741 (+ 345us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:21.686752 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:21.687443 (+ 691us) server_negotiation.cc:407] Connection header received
0806 09:22:21.687971 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:21.687979 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:21.688087 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:21.688172 (+ 85us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:21.689150 (+ 978us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.690575 (+ 1425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.696872 (+ 6297us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.698535 (+ 1663us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.698742 (+ 207us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.698868 (+ 126us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:21.699401 (+ 533us) server_negotiation.cc:299] Negotiation successful
0806 09:22:21.699632 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:22:21.699993 28378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.686339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:46593 (local address 127.16.86.129:39803)
0806 09:22:21.687253 (+ 914us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:21.687282 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:21.687719 (+ 437us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:21.688466 (+ 747us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:21.688472 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:21.688503 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:21.688916 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:21.688930 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.692949 (+ 4019us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.692958 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:21.696546 (+ 3588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:21.696569 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.698877 (+ 2308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.698884 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:21.699175 (+ 291us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.699207 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:22:21.699336 (+ 129us) client_negotiation.cc:240] Negotiation successful
0806 09:22:21.699513 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":633,"thread_start_us":474,"threads_started":1}
I20260806 09:22:21.701664 27964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b41a672b9dd4631ac442a3d06a03475" candidate_uuid: "8eb628a5f0774f82b90d04bd056b5099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773d982788d842f0904c34cf27d30512" is_pre_election: true
I20260806 09:22:21.702188 27964 raft_consensus.cc:2454] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb628a5f0774f82b90d04bd056b5099 in term 0.
I20260806 09:22:21.703078 27666 leader_election.cc:302] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773d982788d842f0904c34cf27d30512, 8eb628a5f0774f82b90d04bd056b5099; no voters:
I20260806 09:22:21.703787 28275 raft_consensus.cc:2781] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:22:21.703991 28275 raft_consensus.cc:482] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:22:21.704155 28275 raft_consensus.cc:3037] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:21.706787 28379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.687347 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:37941 (local address 127.16.86.129:43963)
0806 09:22:21.698702 (+ 11355us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:21.698728 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:21.698888 (+ 160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:21.699652 (+ 764us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:21.699664 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:21.699702 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:21.700171 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:21.700185 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.701672 (+ 1487us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.701680 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:21.703666 (+ 1986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:21.703685 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.705814 (+ 2129us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.705823 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:21.706037 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.706091 (+ 54us) client_negotiation.cc:769] Sending connection context
0806 09:22:21.706207 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 09:22:21.706358 (+ 151us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11190,"thread_start_us":11058,"threads_started":1}
I20260806 09:22:21.707036 28381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.687546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:43963 (local address 127.16.86.131:37941)
0806 09:22:21.698400 (+ 10854us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:21.698409 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:21.698992 (+ 583us) server_negotiation.cc:407] Connection header received
0806 09:22:21.699138 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:21.699144 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:21.699264 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:21.699404 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:21.700388 (+ 984us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.701504 (+ 1116us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.703913 (+ 2409us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.705512 (+ 1599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.705753 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.705859 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:21.706274 (+ 415us) server_negotiation.cc:299] Negotiation successful
0806 09:22:21.706491 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10679,"thread_start_us":10503,"threads_started":1}
I20260806 09:22:21.707401 28275 raft_consensus.cc:504] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.708035 28153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b41a672b9dd4631ac442a3d06a03475" candidate_uuid: "8eb628a5f0774f82b90d04bd056b5099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823c87d9387b401f8142d88e1a746930" is_pre_election: true
I20260806 09:22:21.708379 28275 leader_election.cc:288] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [CANDIDATE]: Term 1 election: Requested vote from peers 773d982788d842f0904c34cf27d30512 (127.16.86.130:46593), 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941)
I20260806 09:22:21.708469 28153 raft_consensus.cc:2454] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8eb628a5f0774f82b90d04bd056b5099 in term 0.
I20260806 09:22:21.709002 27964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b41a672b9dd4631ac442a3d06a03475" candidate_uuid: "8eb628a5f0774f82b90d04bd056b5099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "773d982788d842f0904c34cf27d30512"
I20260806 09:22:21.709348 27964 raft_consensus.cc:3037] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:21.709424 28153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "2b41a672b9dd4631ac442a3d06a03475" candidate_uuid: "8eb628a5f0774f82b90d04bd056b5099" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823c87d9387b401f8142d88e1a746930"
I20260806 09:22:21.709651 28153 raft_consensus.cc:3037] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:21.712378 27964 raft_consensus.cc:2454] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb628a5f0774f82b90d04bd056b5099 in term 1.
I20260806 09:22:21.712891 27666 leader_election.cc:302] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773d982788d842f0904c34cf27d30512, 8eb628a5f0774f82b90d04bd056b5099; no voters:
I20260806 09:22:21.713222 28275 raft_consensus.cc:2781] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:21.713569 28153 raft_consensus.cc:2454] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8eb628a5f0774f82b90d04bd056b5099 in term 1.
I20260806 09:22:21.713917 28275 raft_consensus.cc:686] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 LEADER]: Becoming Leader. State: Replica: 8eb628a5f0774f82b90d04bd056b5099, State: Running, Role: LEADER
I20260806 09:22:21.714432 28275 consensus_queue.cc:237] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.718442 27538 catalog_manager.cc:5495] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129). New cstate: current_term: 1 leader_uuid: "8eb628a5f0774f82b90d04bd056b5099" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } health_report { overall_health: UNKNOWN } } }
I20260806 09:22:21.761636 28387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:21.754493 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43026 (local address 127.16.86.129:35331)
0806 09:22:21.755297 (+ 804us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:21.755302 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:21.755322 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:22:21.755420 (+ 98us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:21.755424 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:21.755515 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:21.755702 (+ 187us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:22:21.756147 (+ 445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.756951 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.758259 (+ 1308us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:21.758895 (+ 636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:21.759080 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:21.760208 (+ 1128us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:22:21.760868 (+ 660us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:22:21.760987 (+ 119us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:21.761221 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 09:22:21.761294 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":672,"thread_start_us":426,"threads_started":1}
I20260806 09:22:21.766237 28153 raft_consensus.cc:1261] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Refusing update from remote peer 8eb628a5f0774f82b90d04bd056b5099: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:21.766361 27964 raft_consensus.cc:1261] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Refusing update from remote peer 8eb628a5f0774f82b90d04bd056b5099: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:21.767556 28275 consensus_queue.cc:1035] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:21.768240 28383 consensus_queue.cc:1035] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [LEADER]: Connected to new peer: Peer: permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:21.778478 28329 mvcc.cc:205] Tried to move back new op lower bound from 7315489348664291328 to 7315489348466855936. Current Snapshot: MvccSnapshot[applied={T|T < 7315489348664291328}]
I20260806 09:22:21.813475 27540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:21.813807 27540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:21.818203 27540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc: Sending DeleteTablet for 3 replicas of tablet 2b41a672b9dd4631ac442a3d06a03475
I20260806 09:22:21.819497 28125 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b41a672b9dd4631ac442a3d06a03475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:21 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:40686
I20260806 09:22:21.819730 27930 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b41a672b9dd4631ac442a3d06a03475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:21 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:34080
I20260806 09:22:21.820225 27730 tablet_service.cc:1514] Processing DeleteTablet for tablet 2b41a672b9dd4631ac442a3d06a03475 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:21 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:43010
I20260806 09:22:21.820626 28351 tablet_replica.cc:330] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930: stopping tablet replica
I20260806 09:22:21.820477 27540 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34882:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20260806 09:22:21.820906 28351 raft_consensus.cc:2229] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Raft consensus shutting down.
W20260806 09:22:21.821180 27540 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:22:21.821223 28351 raft_consensus.cc:2258] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:21.821617 28353 tablet_replica.cc:330] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512: stopping tablet replica
I20260806 09:22:21.821888 28353 raft_consensus.cc:2229] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:21.822223 28353 raft_consensus.cc:2258] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:21.822599 28350 tablet_replica.cc:330] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099: stopping tablet replica
I20260806 09:22:21.822861 28350 raft_consensus.cc:2229] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:22:21.824721 28350 raft_consensus.cc:2258] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:21.828001 28351 ts_tablet_manager.cc:1905] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:21.832063 28353 ts_tablet_manager.cc:1905] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:21.836820 28350 ts_tablet_manager.cc:1905] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:21.855473 28125 tablet_service.cc:1467] Processing CreateTablet for tablet dd159414cb81463d937d71025b896ab9 (DEFAULT_TABLE table=test-table [id=60e18d02019646edb2c5b0eca8fdbc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:21.856159 28125 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd159414cb81463d937d71025b896ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:21.860452 27730 tablet_service.cc:1467] Processing CreateTablet for tablet dd159414cb81463d937d71025b896ab9 (DEFAULT_TABLE table=test-table [id=60e18d02019646edb2c5b0eca8fdbc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:21.861064 27730 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd159414cb81463d937d71025b896ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:21.864429 27930 tablet_service.cc:1467] Processing CreateTablet for tablet dd159414cb81463d937d71025b896ab9 (DEFAULT_TABLE table=test-table [id=60e18d02019646edb2c5b0eca8fdbc3d]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:21.865046 27930 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet dd159414cb81463d937d71025b896ab9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:21.866102 28353 ts_tablet_manager.cc:1918] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:21.866377 28353 log.cc:1196] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/2b41a672b9dd4631ac442a3d06a03475
I20260806 09:22:21.867028 28351 ts_tablet_manager.cc:1918] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:21.867199 28351 log.cc:1196] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/2b41a672b9dd4631ac442a3d06a03475
I20260806 09:22:21.867835 28351 ts_tablet_manager.cc:1939] T 2b41a672b9dd4631ac442a3d06a03475 P 823c87d9387b401f8142d88e1a746930: Deleting consensus metadata
I20260806 09:22:21.869355 27520 catalog_manager.cc:4841] TS 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941): tablet 2b41a672b9dd4631ac442a3d06a03475 (table test-table [id=d78bcb9c20d14fbbb36768677ef2310e]) successfully deleted
I20260806 09:22:21.870244 28353 ts_tablet_manager.cc:1939] T 2b41a672b9dd4631ac442a3d06a03475 P 773d982788d842f0904c34cf27d30512: Deleting consensus metadata
I20260806 09:22:21.871752 27520 catalog_manager.cc:4841] TS 773d982788d842f0904c34cf27d30512 (127.16.86.130:46593): tablet 2b41a672b9dd4631ac442a3d06a03475 (table test-table [id=d78bcb9c20d14fbbb36768677ef2310e]) successfully deleted
I20260806 09:22:21.873430 28374 tablet_bootstrap.cc:492] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512: Bootstrap starting.
I20260806 09:22:21.875681 28374 tablet_bootstrap.cc:654] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:21.876850 28373 tablet_bootstrap.cc:492] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930: Bootstrap starting.
I20260806 09:22:21.877903 28350 ts_tablet_manager.cc:1918] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:21.878098 28350 log.cc:1196] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/2b41a672b9dd4631ac442a3d06a03475
I20260806 09:22:21.878590 28350 ts_tablet_manager.cc:1939] T 2b41a672b9dd4631ac442a3d06a03475 P 8eb628a5f0774f82b90d04bd056b5099: Deleting consensus metadata
I20260806 09:22:21.878794 28373 tablet_bootstrap.cc:654] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:21.879886 27521 catalog_manager.cc:4841] TS 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331): tablet 2b41a672b9dd4631ac442a3d06a03475 (table test-table [id=d78bcb9c20d14fbbb36768677ef2310e]) successfully deleted
I20260806 09:22:21.886440 28372 tablet_bootstrap.cc:492] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099: Bootstrap starting.
I20260806 09:22:21.888420 28372 tablet_bootstrap.cc:654] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:21.891377 28372 tablet_bootstrap.cc:492] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099: No bootstrap required, opened a new log
I20260806 09:22:21.891732 28372 ts_tablet_manager.cc:1397] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260806 09:22:21.893000 28372 raft_consensus.cc:348] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.893481 28372 raft_consensus.cc:374] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:21.895339 28372 raft_consensus.cc:729] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8eb628a5f0774f82b90d04bd056b5099, State: Initialized, Role: FOLLOWER
I20260806 09:22:21.895855 28372 consensus_queue.cc:260] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.900619 28372 ts_tablet_manager.cc:1428] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260806 09:22:21.900812 28374 tablet_bootstrap.cc:492] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512: No bootstrap required, opened a new log
I20260806 09:22:21.901034 28374 ts_tablet_manager.cc:1397] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512: Time spent bootstrapping tablet: real 0.028s user 0.005s sys 0.007s
I20260806 09:22:21.902043 28374 raft_consensus.cc:348] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.902360 28374 raft_consensus.cc:374] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:21.902464 28374 raft_consensus.cc:729] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 773d982788d842f0904c34cf27d30512, State: Initialized, Role: FOLLOWER
I20260806 09:22:21.904366 28373 tablet_bootstrap.cc:492] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930: No bootstrap required, opened a new log
I20260806 09:22:21.904603 28373 ts_tablet_manager.cc:1397] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930: Time spent bootstrapping tablet: real 0.028s user 0.007s sys 0.005s
I20260806 09:22:21.905608 28373 raft_consensus.cc:348] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.905915 28373 raft_consensus.cc:374] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:21.906018 28373 raft_consensus.cc:729] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 823c87d9387b401f8142d88e1a746930, State: Initialized, Role: FOLLOWER
I20260806 09:22:21.905699 28374 consensus_queue.cc:260] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.906679 28374 ts_tablet_manager.cc:1428] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512: Time spent starting tablet: real 0.006s user 0.002s sys 0.001s
I20260806 09:22:21.906337 28373 consensus_queue.cc:260] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:21.907721 28373 ts_tablet_manager.cc:1428] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260806 09:22:22.099840 28336 raft_consensus.cc:482] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:22:22.100119 28336 raft_consensus.cc:504] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:22.101006 28336 leader_election.cc:288] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331), 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941)
I20260806 09:22:22.101969 27765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd159414cb81463d937d71025b896ab9" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb628a5f0774f82b90d04bd056b5099" is_pre_election: true
I20260806 09:22:22.102025 28153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd159414cb81463d937d71025b896ab9" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823c87d9387b401f8142d88e1a746930" is_pre_election: true
I20260806 09:22:22.102368 28153 raft_consensus.cc:2454] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 0.
I20260806 09:22:22.102375 27765 raft_consensus.cc:2454] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 0.
I20260806 09:22:22.103057 27863 leader_election.cc:302] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773d982788d842f0904c34cf27d30512, 8eb628a5f0774f82b90d04bd056b5099; no voters:
I20260806 09:22:22.103459 28336 raft_consensus.cc:2781] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:22:22.103614 28336 raft_consensus.cc:482] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:22:22.103724 28336 raft_consensus.cc:3037] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:22.107071 28336 raft_consensus.cc:504] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:22.107944 28336 leader_election.cc:288] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 election: Requested vote from peers 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331), 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941)
I20260806 09:22:22.108925 27765 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd159414cb81463d937d71025b896ab9" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8eb628a5f0774f82b90d04bd056b5099"
I20260806 09:22:22.108920 28153 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "dd159414cb81463d937d71025b896ab9" candidate_uuid: "773d982788d842f0904c34cf27d30512" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "823c87d9387b401f8142d88e1a746930"
I20260806 09:22:22.109221 27765 raft_consensus.cc:3037] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:22.109265 28153 raft_consensus.cc:3037] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:22.112293 28153 raft_consensus.cc:2454] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 1.
I20260806 09:22:22.112780 27765 raft_consensus.cc:2454] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 773d982788d842f0904c34cf27d30512 in term 1.
I20260806 09:22:22.113060 27862 leader_election.cc:302] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 773d982788d842f0904c34cf27d30512, 823c87d9387b401f8142d88e1a746930; no voters:
I20260806 09:22:22.113628 28336 raft_consensus.cc:2781] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:22.114656 28336 raft_consensus.cc:686] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 1 LEADER]: Becoming Leader. State: Replica: 773d982788d842f0904c34cf27d30512, State: Running, Role: LEADER
I20260806 09:22:22.115073 28336 consensus_queue.cc:237] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } }
I20260806 09:22:22.118685 27538 catalog_manager.cc:5495] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 reported cstate change: term changed from 0 to 1, leader changed from <none> to 773d982788d842f0904c34cf27d30512 (127.16.86.130). New cstate: current_term: 1 leader_uuid: "773d982788d842f0904c34cf27d30512" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "773d982788d842f0904c34cf27d30512" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46593 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 } health_report { overall_health: UNKNOWN } } }
I20260806 09:22:22.199548 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:22.192324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34114 (local address 127.16.86.130:46593)
0806 09:22:22.192579 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:22.192587 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:22.192779 (+ 192us) server_negotiation.cc:407] Connection header received
0806 09:22:22.192971 (+ 192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:22.192977 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:22.193097 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:22.193236 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:22:22.193682 (+ 446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:22.194904 (+ 1222us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:22.197317 (+ 2413us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:22.197959 (+ 642us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:22.198134 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:22.198616 (+ 482us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:22:22.198771 (+ 155us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:22:22.198885 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:22.199106 (+ 221us) server_negotiation.cc:299] Negotiation successful
0806 09:22:22.199220 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 09:22:22.213677 27765 raft_consensus.cc:1261] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Refusing update from remote peer 773d982788d842f0904c34cf27d30512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:22.214334 28407 consensus_queue.cc:1035] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8eb628a5f0774f82b90d04bd056b5099" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35331 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:22.217437 28153 raft_consensus.cc:1261] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Refusing update from remote peer 773d982788d842f0904c34cf27d30512: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:22.218169 28407 consensus_queue.cc:1035] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [LEADER]: Connected to new peer: Peer: permanent_uuid: "823c87d9387b401f8142d88e1a746930" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 37941 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:22.305095 27540 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:22.305409 27540 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34882:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:22.309468 27540 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ec383fe1ca2430fb73364f156a803fc: Sending DeleteTablet for 3 replicas of tablet dd159414cb81463d937d71025b896ab9
I20260806 09:22:22.310639 27730 tablet_service.cc:1514] Processing DeleteTablet for tablet dd159414cb81463d937d71025b896ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:22 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:43010
I20260806 09:22:22.311060 28350 tablet_replica.cc:330] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099: stopping tablet replica
I20260806 09:22:22.311033 28125 tablet_service.cc:1514] Processing DeleteTablet for tablet dd159414cb81463d937d71025b896ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:22 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:40686
I20260806 09:22:22.311307 28350 raft_consensus.cc:2229] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:22.311394 28351 tablet_replica.cc:330] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930: stopping tablet replica
I20260806 09:22:22.311546 28350 raft_consensus.cc:2258] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:22.311551 28351 raft_consensus.cc:2229] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:22.311777 28351 raft_consensus.cc:2258] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:22.313023 28350 ts_tablet_manager.cc:1905] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:22.313513 28351 ts_tablet_manager.cc:1905] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:22.319404 27930 tablet_service.cc:1514] Processing DeleteTablet for tablet dd159414cb81463d937d71025b896ab9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:22 UTC) from {username='oryx', principal='oryx/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:34080
I20260806 09:22:22.318938 28351 ts_tablet_manager.cc:1918] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:22:22.319854 28351 log.cc:1196] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/dd159414cb81463d937d71025b896ab9
I20260806 09:22:22.320217 28350 ts_tablet_manager.cc:1918] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:22:22.320412 28350 log.cc:1196] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/dd159414cb81463d937d71025b896ab9
I20260806 09:22:22.320640 28351 ts_tablet_manager.cc:1939] T dd159414cb81463d937d71025b896ab9 P 823c87d9387b401f8142d88e1a746930: Deleting consensus metadata
I20260806 09:22:22.320963 28350 ts_tablet_manager.cc:1939] T dd159414cb81463d937d71025b896ab9 P 8eb628a5f0774f82b90d04bd056b5099: Deleting consensus metadata
I20260806 09:22:22.322384 27520 catalog_manager.cc:4841] TS 823c87d9387b401f8142d88e1a746930 (127.16.86.131:37941): tablet dd159414cb81463d937d71025b896ab9 (table test-table [id=60e18d02019646edb2c5b0eca8fdbc3d]) successfully deleted
I20260806 09:22:22.322546 27521 catalog_manager.cc:4841] TS 8eb628a5f0774f82b90d04bd056b5099 (127.16.86.129:35331): tablet dd159414cb81463d937d71025b896ab9 (table test-table [id=60e18d02019646edb2c5b0eca8fdbc3d]) successfully deleted
I20260806 09:22:22.323194 28353 tablet_replica.cc:330] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512: stopping tablet replica
I20260806 09:22:22.323447 28353 raft_consensus.cc:2229] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:22:22.323885 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 27644
I20260806 09:22:22.323964 28353 raft_consensus.cc:2258] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:22.325510 28353 ts_tablet_manager.cc:1905] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:22.336704 28353 ts_tablet_manager.cc:1918] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20260806 09:22:22.336943 28353 log.cc:1196] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/dd159414cb81463d937d71025b896ab9
I20260806 09:22:22.337517 28353 ts_tablet_manager.cc:1939] T dd159414cb81463d937d71025b896ab9 P 773d982788d842f0904c34cf27d30512: Deleting consensus metadata
I20260806 09:22:22.339214 27520 catalog_manager.cc:4841] TS 773d982788d842f0904c34cf27d30512 (127.16.86.130:46593): tablet dd159414cb81463d937d71025b896ab9 (table test-table [id=60e18d02019646edb2c5b0eca8fdbc3d]) successfully deleted
Aug 06 09:22:22 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.86.129@KRBTEST.COM
I20260806 09:22:22.363286 28414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:22.325935 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43040 (local address 127.16.86.129:35331)
0806 09:22:22.340001 (+ 14066us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:22.340010 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:22.340038 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:22:22.340119 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:22.340124 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:22.340215 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:22.340355 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:22.341882 (+ 1527us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:22.343037 (+ 1155us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:22.344440 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:22.345143 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:22.345291 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:22.355838 (+ 10547us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:22.355858 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:22.355876 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:22.355925 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:22.359297 (+ 3372us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:22.360265 (+ 968us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:22.360277 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:22.360280 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:22.360646 (+ 366us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:22.361144 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:22.361150 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:22.361154 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:22.361660 (+ 506us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:22.362081 (+ 421us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:22.362408 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 09:22:22.362742 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13843,"thread_start_us":13644,"threads_started":1}
I20260806 09:22:22.364205 27821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:22.514359 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 27644
I20260806 09:22:22.554659 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 27833
Aug 06 09:22:22 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.86.130@KRBTEST.COM
I20260806 09:22:22.574389 28324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:22.555787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34118 (local address 127.16.86.130:46593)
0806 09:22:22.555974 (+ 187us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:22.555978 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:22.555998 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:22:22.556116 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:22.556120 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:22.556189 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:22.556303 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:22.558101 (+ 1798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:22.558902 (+ 801us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:22.560774 (+ 1872us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:22.561171 (+ 397us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:22.561274 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:22.567297 (+ 6023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:22.567314 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:22.567331 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:22.567374 (+ 43us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:22.570898 (+ 3524us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:22.571957 (+ 1059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:22.571965 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:22.571967 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:22.572145 (+ 178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:22.572680 (+ 535us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:22.572684 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:22.572687 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:22.573109 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:22.573416 (+ 307us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:22.573871 (+ 455us) server_negotiation.cc:299] Negotiation successful
0806 09:22:22.574120 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:22:22.575629 28020 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:22.735146 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 27833
W20260806 09:22:22.770768 28060 connection.cc:535] server connection from 127.16.86.130:43071 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 09:22:22.771411 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 28032
Aug 06 09:22:22 dist-test-slave-h3ft krb5kdc[27484](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008141, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.16.86.131@KRBTEST.COM
I20260806 09:22:22.789798 28419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:22.772470 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40698 (local address 127.16.86.131:37941)
0806 09:22:22.773102 (+ 632us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:22.773107 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:22.773130 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:22:22.773205 (+ 75us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:22.773210 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:22.773288 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:22.773412 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:22.774893 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:22.775771 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:22.777244 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:22.777706 (+ 462us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:22.777809 (+ 103us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:22.782692 (+ 4883us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:22.782715 (+ 23us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:22.782738 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:22.782791 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:22.786589 (+ 3798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:22.787545 (+ 956us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:22.787553 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:22.787556 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:22.787735 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:22.788167 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:22.788171 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:22.788173 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:22.788613 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:22.788891 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:22.789233 (+ 342us) server_negotiation.cc:299] Negotiation successful
0806 09:22:22.789457 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":500,"thread_start_us":340,"threads_started":1}
I20260806 09:22:22.790719 28221 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:22.907702 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 28032
I20260806 09:22:22.936476 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 27499
I20260806 09:22:22.938059 27601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:23.062700 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 27499
2026-08-06T09:22:23Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (8356 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:22:23 dist-test-slave-h3ft krb5kdc[28428](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:22:23 dist-test-slave-h3ft krb5kdc[28428](info): set up 2 sockets
Aug 06 09:22:23 dist-test-slave-h3ft krb5kdc[28428](info): commencing operation
krb5kdc: starting...
W20260806 09:22:24.406790 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.281s user 0.001s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:22:24 dist-test-slave-h3ft krb5kdc[28428](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008144, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:22:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:22:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:22:24.755861 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:38215
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:43669
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:38215
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:25.156143 28450 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:25.156546 28450 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:25.156740 28450 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:25.164207 28450 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:22:25.164366 28450 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:25.164484 28450 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:25.164610 28450 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:22:25.164708 28450 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:22:25.174923 28450 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:43669
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:38215
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:38215
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:25.176447 28450 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:25.177878 28450 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:25.191635 28457 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:25.191716 28458 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:25.191744 28462 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:25.192845 28450 server_base.cc:1034] running on GCE node
I20260806 09:22:25.193773 28450 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:25.195384 28450 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:25.196682 28450 hybrid_clock.cc:648] HybridClock initialized: now 1786008145196599 us; error 55 us; skew 500 ppm
I20260806 09:22:25.197095 28450 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2026-08-06T09:22:25Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2195 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:22:25 dist-test-slave-h3ft krb5kdc[28471](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:22:25 dist-test-slave-h3ft krb5kdc[28471](info): set up 2 sockets
Aug 06 09:22:25 dist-test-slave-h3ft krb5kdc[28471](info): commencing operation
krb5kdc: starting...
W20260806 09:22:26.653155 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.333s user 0.000s sys 0.044s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:22:26 dist-test-slave-h3ft krb5kdc[28471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:22:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:22:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:22:27.000785 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41485
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:41485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:27.416080 28486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:27.416486 28486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:27.416723 28486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:27.425515 28486 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:22:27.425685 28486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:27.425815 28486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:27.425949 28486 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:22:27.426051 28486 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:22:27.438644 28486 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:41485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:27.440668 28486 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:27.442430 28486 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:27.458029 28493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:27.458166 28496 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:27.458149 28494 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:27.460799 28486 server_base.cc:1034] running on GCE node
I20260806 09:22:27.461908 28486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:27.464263 28486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:27.465582 28486 hybrid_clock.cc:648] HybridClock initialized: now 1786008147465515 us; error 66 us; skew 500 ppm
I20260806 09:22:27.466215 28486 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:27 dist-test-slave-h3ft krb5kdc[28471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008147, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:27.474534 28486 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:22:27.476900 28486 webserver.cc:460] Webserver started at http://127.16.86.190:37095/ using document root <none> and password file <none>
I20260806 09:22:27.477859 28486 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:27.478044 28486 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:27.478433 28486 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:27.482143 28486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "4ced673db7f2454a9cdabd60cdafddc5"
format_stamp: "Formatted at 2026-08-06 09:22:27 on dist-test-slave-h3ft"
server_key: "17afbfa9e9315af04ec4f9652f132b0c"
server_key_iv: "1fe80e7e6bd4251c2a7ab59820efc877"
server_key_version: "encryptionkey@0"
I20260806 09:22:27.483130 28486 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "4ced673db7f2454a9cdabd60cdafddc5"
format_stamp: "Formatted at 2026-08-06 09:22:27 on dist-test-slave-h3ft"
server_key: "17afbfa9e9315af04ec4f9652f132b0c"
server_key_iv: "1fe80e7e6bd4251c2a7ab59820efc877"
server_key_version: "encryptionkey@0"
I20260806 09:22:27.490077 28486 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:22:27.495128 28506 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:27.497041 28486 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:22:27.497318 28486 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "4ced673db7f2454a9cdabd60cdafddc5"
format_stamp: "Formatted at 2026-08-06 09:22:27 on dist-test-slave-h3ft"
server_key: "17afbfa9e9315af04ec4f9652f132b0c"
server_key_iv: "1fe80e7e6bd4251c2a7ab59820efc877"
server_key_version: "encryptionkey@0"
I20260806 09:22:27.497567 28486 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:27.527362 28486 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:27.532704 28486 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:27.533064 28486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:27.557161 28486 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:41485
I20260806 09:22:27.557214 28584 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:41485 every 8 connection(s)
I20260806 09:22:27.558878 28486 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:22:27.564080 28585 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:27.567143 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 28486
I20260806 09:22:27.567431 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:22:27.568364 16730 external_mini_cluster.cc:1428] Setting key 3d859583c31b70da64eed34f05390126
I20260806 09:22:27.580341 28585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5: Bootstrap starting.
I20260806 09:22:27.585446 28585 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:27.587164 28585 log.cc:824] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:22:27 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:22:27.591966 28585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5: No bootstrap required, opened a new log
I20260806 09:22:27.600231 28588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:27.571486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60682 (local address 127.16.86.190:41485)
0806 09:22:27.572581 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:27.572595 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:27.572666 (+ 71us) server_negotiation.cc:407] Connection header received
0806 09:22:27.574873 (+ 2207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:27.574900 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:27.575505 (+ 605us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:27.575941 (+ 436us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:27.578306 (+ 2365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:27.580015 (+ 1709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:27.582004 (+ 1989us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:27.582901 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:27.583165 (+ 264us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:27.589420 (+ 6255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:27.589435 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:27.589461 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:27.589528 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:27.594128 (+ 4600us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:27.595101 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:27.595111 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:27.595122 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:27.595414 (+ 292us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:27.595961 (+ 547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:27.595967 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:27.595971 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:27.596905 (+ 934us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:27.597270 (+ 365us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:27.597749 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 09:22:27.598383 (+ 634us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":805,"thread_start_us":516,"threads_started":1}
I20260806 09:22:27.603964 28585 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced673db7f2454a9cdabd60cdafddc5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41485 } }
I20260806 09:22:27.604527 28585 raft_consensus.cc:374] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:27.604624 28585 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ced673db7f2454a9cdabd60cdafddc5, State: Initialized, Role: FOLLOWER
I20260806 09:22:27.605195 28585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced673db7f2454a9cdabd60cdafddc5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41485 } }
I20260806 09:22:27.605433 28585 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:22:27.605545 28585 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:22:27.605690 28585 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:27.608248 28585 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced673db7f2454a9cdabd60cdafddc5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41485 } }
I20260806 09:22:27.608855 28585 leader_election.cc:302] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ced673db7f2454a9cdabd60cdafddc5; no voters:
I20260806 09:22:27.609701 28585 leader_election.cc:288] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:22:27.609829 28590 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:27.610798 28590 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 1 LEADER]: Becoming Leader. State: Replica: 4ced673db7f2454a9cdabd60cdafddc5, State: Running, Role: LEADER
I20260806 09:22:27.611572 28590 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced673db7f2454a9cdabd60cdafddc5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41485 } }
I20260806 09:22:27.613005 28585 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:22:27.615202 28593 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4ced673db7f2454a9cdabd60cdafddc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced673db7f2454a9cdabd60cdafddc5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41485 } } }
I20260806 09:22:27.615339 28592 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ced673db7f2454a9cdabd60cdafddc5. Latest consensus state: current_term: 1 leader_uuid: "4ced673db7f2454a9cdabd60cdafddc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced673db7f2454a9cdabd60cdafddc5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41485 } } }
I20260806 09:22:27.615741 28593 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:27.615837 28592 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:27.620504 28598 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:22:27.627156 28598 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:22:27.638181 28598 catalog_manager.cc:1347] Generated new cluster ID: 09bf2af412574d3dab7cce71046078cc
I20260806 09:22:27.638367 28598 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:22:27.653461 28598 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:22:27.655247 28598 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:22:27.669574 28598 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5: Generated new TSK 0
I20260806 09:22:27.670727 28598 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:22:27.756760 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41485
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:28.188097 28622 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:28.188482 28622 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:28.188666 28622 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:28.196712 28622 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:28.196866 28622 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:28.197077 28622 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:22:28.208369 28622 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:28.210152 28622 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:28.211875 28622 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:28.229121 28629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:28.229244 28630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:28.229707 28634 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:28.230350 28622 server_base.cc:1034] running on GCE node
I20260806 09:22:28.231174 28622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:28.232359 28622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:28.233613 28622 hybrid_clock.cc:648] HybridClock initialized: now 1786008148233521 us; error 76 us; skew 500 ppm
I20260806 09:22:28.233984 28622 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:28 dist-test-slave-h3ft krb5kdc[28471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008148, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:28.238517 28622 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:22:28.240566 28622 webserver.cc:460] Webserver started at http://127.16.86.129:42873/ using document root <none> and password file <none>
I20260806 09:22:28.241472 28622 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:28.241587 28622 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:28.241902 28622 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:28.245144 28622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "307c4ad1acc747ad8b4d29acfb6b6b39"
format_stamp: "Formatted at 2026-08-06 09:22:28 on dist-test-slave-h3ft"
server_key: "a9ebcdafb0f6f51606ee9c506f12b961"
server_key_iv: "1f0bf54f5663ac35e27c88d154349497"
server_key_version: "encryptionkey@0"
I20260806 09:22:28.245947 28622 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "307c4ad1acc747ad8b4d29acfb6b6b39"
format_stamp: "Formatted at 2026-08-06 09:22:28 on dist-test-slave-h3ft"
server_key: "a9ebcdafb0f6f51606ee9c506f12b961"
server_key_iv: "1f0bf54f5663ac35e27c88d154349497"
server_key_version: "encryptionkey@0"
I20260806 09:22:28.252161 28622 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 09:22:28.256518 28642 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:28.258262 28622 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:22:28.258451 28622 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "307c4ad1acc747ad8b4d29acfb6b6b39"
format_stamp: "Formatted at 2026-08-06 09:22:28 on dist-test-slave-h3ft"
server_key: "a9ebcdafb0f6f51606ee9c506f12b961"
server_key_iv: "1f0bf54f5663ac35e27c88d154349497"
server_key_version: "encryptionkey@0"
I20260806 09:22:28.258648 28622 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:28.275789 28622 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:28.280498 28622 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:28.280776 28622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:28.281960 28622 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:28.283761 28622 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:28.283844 28622 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:28.284005 28622 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:28.284088 28622 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:28.326609 28622 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:44991
I20260806 09:22:28.326649 28798 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:44991 every 8 connection(s)
I20260806 09:22:28.328114 28622 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:22:28.334004 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 28622
I20260806 09:22:28.334263 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:22:28.335072 16730 external_mini_cluster.cc:1428] Setting key 83c1e7859adcdf3c2cc4b67a4538934b
Aug 06 09:22:28 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008148, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:22:28.351567 28801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:28.331519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:57065 (local address 127.16.86.190:41485)
0806 09:22:28.332080 (+ 561us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:28.332086 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:28.333362 (+ 1276us) server_negotiation.cc:407] Connection header received
0806 09:22:28.335643 (+ 2281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:28.335649 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:28.335742 (+ 93us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:28.335877 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:28.337799 (+ 1922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:28.338538 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:28.340244 (+ 1706us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:28.340663 (+ 419us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:28.340809 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:28.344456 (+ 3647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:28.344470 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:28.344473 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:28.344507 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:28.347285 (+ 2778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:28.348209 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:28.348214 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:28.348216 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:28.348478 (+ 262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:28.349045 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:28.349052 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:28.349056 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:28.349407 (+ 351us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:28.349532 (+ 125us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:28.351069 (+ 1537us) server_negotiation.cc:299] Negotiation successful
0806 09:22:28.351284 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":432,"thread_start_us":290,"threads_started":1}
I20260806 09:22:28.353147 28802 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:28.332241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:57065)
0806 09:22:28.333153 (+ 912us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:28.333213 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:28.335340 (+ 2127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:28.336150 (+ 810us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:28.336161 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:28.336950 (+ 789us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:28.337602 (+ 652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:28.337620 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:28.338725 (+ 1105us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:28.338729 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:28.340079 (+ 1350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:28.340093 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:28.340915 (+ 822us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:28.340925 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:28.341186 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:28.341938 (+ 752us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:28.341962 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:28.344221 (+ 2259us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:28.347434 (+ 3213us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:28.347443 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:28.347461 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:28.348051 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:28.348618 (+ 567us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:28.348623 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:28.348629 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:28.348901 (+ 272us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:28.349649 (+ 748us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:28.349665 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:28.350394 (+ 729us) client_negotiation.cc:769] Sending connection context
0806 09:22:28.350860 (+ 466us) client_negotiation.cc:240] Negotiation successful
0806 09:22:28.351705 (+ 845us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":621,"thread_start_us":367,"threads_started":1}
I20260806 09:22:28.355271 28799 heartbeater.cc:347] Connected to a master server at 127.16.86.190:41485
I20260806 09:22:28.355674 28799 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:28.356609 28799 heartbeater.cc:511] Master 127.16.86.190:41485 requested a full tablet report, sending...
I20260806 09:22:28.359305 28532 ts_manager.cc:194] Registered new tserver with Master: 307c4ad1acc747ad8b4d29acfb6b6b39 (127.16.86.129:44991)
I20260806 09:22:28.361132 28532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:57065
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:22:28.407838 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41485
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:28.800226 28807 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:28.800647 28807 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:28.800849 28807 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:28.810025 28807 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:28.810402 28807 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:28.810621 28807 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:22:28.821614 28807 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:28.823115 28807 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:28.824532 28807 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:28.839619 28820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:28.839704 28817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:28.840044 28816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:28.841074 28807 server_base.cc:1034] running on GCE node
I20260806 09:22:28.842006 28807 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:28.851424 28807 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:28.855072 28807 hybrid_clock.cc:648] HybridClock initialized: now 1786008148854913 us; error 120 us; skew 500 ppm
I20260806 09:22:28.855726 28807 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:28 dist-test-slave-h3ft krb5kdc[28471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008148, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:28.861030 28807 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:22:28.863444 28807 webserver.cc:460] Webserver started at http://127.16.86.130:41479/ using document root <none> and password file <none>
I20260806 09:22:28.864483 28807 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:28.864665 28807 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:28.865095 28807 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:28.869130 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "9597eea916dd47ccad033ae3931d4c96"
format_stamp: "Formatted at 2026-08-06 09:22:28 on dist-test-slave-h3ft"
server_key: "fcca3276b3335fc7fbd44068b062dfc3"
server_key_iv: "1394df0576961562f5257a723843ad6a"
server_key_version: "encryptionkey@0"
I20260806 09:22:28.870548 28807 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "9597eea916dd47ccad033ae3931d4c96"
format_stamp: "Formatted at 2026-08-06 09:22:28 on dist-test-slave-h3ft"
server_key: "fcca3276b3335fc7fbd44068b062dfc3"
server_key_iv: "1394df0576961562f5257a723843ad6a"
server_key_version: "encryptionkey@0"
I20260806 09:22:28.878290 28807 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.002s
I20260806 09:22:28.883483 28828 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:28.885499 28807 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:22:28.885804 28807 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "9597eea916dd47ccad033ae3931d4c96"
format_stamp: "Formatted at 2026-08-06 09:22:28 on dist-test-slave-h3ft"
server_key: "fcca3276b3335fc7fbd44068b062dfc3"
server_key_iv: "1394df0576961562f5257a723843ad6a"
server_key_version: "encryptionkey@0"
I20260806 09:22:28.886149 28807 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:28.925200 28807 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:28.931586 28807 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:28.932049 28807 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:28.933539 28807 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:28.935763 28807 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:28.935917 28807 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:28.936124 28807 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:28.936236 28807 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:28.983911 28807 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:35661
I20260806 09:22:28.983995 28995 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:35661 every 8 connection(s)
I20260806 09:22:28.985838 28807 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:22:28.986171 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 28807
I20260806 09:22:28.986377 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:22:28.986912 16730 external_mini_cluster.cc:1428] Setting key d6e0185c991975edd1fe6a429a48f5e9
Aug 06 09:22:29 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008148, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:22:29.022871 28999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:28.989810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:39369 (local address 127.16.86.190:41485)
0806 09:22:28.997907 (+ 8097us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:28.997912 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:28.997938 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:22:29.000817 (+ 2879us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:29.000823 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:29.000931 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:29.001093 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:29.004420 (+ 3327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:29.005301 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:29.007962 (+ 2661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:29.008474 (+ 512us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:29.008644 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:29.013704 (+ 5060us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:29.013715 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:29.013724 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:29.013795 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:29.018215 (+ 4420us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:29.019519 (+ 1304us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:29.019528 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:29.019534 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:29.019726 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:29.020396 (+ 670us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:29.020403 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:29.020410 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:29.020884 (+ 474us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:29.021077 (+ 193us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:29.022098 (+ 1021us) server_negotiation.cc:299] Negotiation successful
0806 09:22:29.022396 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7962,"thread_start_us":7822,"threads_started":1}
I20260806 09:22:29.024266 29001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:28.990824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:39369)
0806 09:22:28.996629 (+ 5805us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:28.996741 (+ 112us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:29.000181 (+ 3440us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:29.001617 (+ 1436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:29.001639 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:29.002762 (+ 1123us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:29.004059 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:29.004101 (+ 42us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:29.005552 (+ 1451us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:29.005561 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:29.007708 (+ 2147us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:29.007725 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:29.008792 (+ 1067us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:29.008800 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:29.009111 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:29.010014 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:29.010041 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:29.013254 (+ 3213us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:29.018459 (+ 5205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:29.018470 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:29.018491 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:29.019326 (+ 835us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:29.019885 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:29.019889 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:29.019892 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:29.020239 (+ 347us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:29.021053 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:29.021060 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:29.021587 (+ 527us) client_negotiation.cc:769] Sending connection context
0806 09:22:29.021945 (+ 358us) client_negotiation.cc:240] Negotiation successful
0806 09:22:29.022542 (+ 597us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5257,"thread_start_us":4872,"threads_started":1}
I20260806 09:22:29.026196 28996 heartbeater.cc:347] Connected to a master server at 127.16.86.190:41485
I20260806 09:22:29.026588 28996 heartbeater.cc:464] Registering TS with master...
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
I20260806 09:22:29.027577 28996 heartbeater.cc:511] Master 127.16.86.190:41485 requested a full tablet report, sending...
I20260806 09:22:29.029734 28532 ts_manager.cc:194] Registered new tserver with Master: 9597eea916dd47ccad033ae3931d4c96 (127.16.86.130:35661)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
I20260806 09:22:29.030859 28532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:39369
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:22:29.068157 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41485
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:22:29.365207 28799 heartbeater.cc:503] Master 127.16.86.190:41485 was elected leader, sending a full tablet report...
W20260806 09:22:29.446064 29005 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:29.446376 29005 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:29.446520 29005 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:29.454084 29005 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:29.454212 29005 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:29.454388 29005 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:22:29.465337 29005 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:29.466706 29005 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:29.468125 29005 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:29.488615 29011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:29.488838 29010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:29.488787 29013 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:29.489259 29005 server_base.cc:1034] running on GCE node
I20260806 09:22:29.490331 29005 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:29.491523 29005 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:29.492759 29005 hybrid_clock.cc:648] HybridClock initialized: now 1786008149492700 us; error 42 us; skew 500 ppm
I20260806 09:22:29.493105 29005 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:29 dist-test-slave-h3ft krb5kdc[28471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008149, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:29.496623 29005 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:22:29.498416 29005 webserver.cc:460] Webserver started at http://127.16.86.131:39893/ using document root <none> and password file <none>
I20260806 09:22:29.499228 29005 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:29.499373 29005 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:29.499722 29005 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:29.502717 29005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "e64dc4e536104f17ab81d8df05e76e14"
format_stamp: "Formatted at 2026-08-06 09:22:29 on dist-test-slave-h3ft"
server_key: "5ac8219c266709668ca695c6fa75c4b9"
server_key_iv: "8bec0b0dc8c9fb7d8497160b23d6e531"
server_key_version: "encryptionkey@0"
I20260806 09:22:29.503674 29005 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "e64dc4e536104f17ab81d8df05e76e14"
format_stamp: "Formatted at 2026-08-06 09:22:29 on dist-test-slave-h3ft"
server_key: "5ac8219c266709668ca695c6fa75c4b9"
server_key_iv: "8bec0b0dc8c9fb7d8497160b23d6e531"
server_key_version: "encryptionkey@0"
I20260806 09:22:29.509414 29005 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:22:29.513677 29026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:29.515619 29005 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:22:29.515853 29005 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "e64dc4e536104f17ab81d8df05e76e14"
format_stamp: "Formatted at 2026-08-06 09:22:29 on dist-test-slave-h3ft"
server_key: "5ac8219c266709668ca695c6fa75c4b9"
server_key_iv: "8bec0b0dc8c9fb7d8497160b23d6e531"
server_key_version: "encryptionkey@0"
I20260806 09:22:29.516076 29005 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:29.534444 29005 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:29.539565 29005 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:29.539893 29005 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:29.541065 29005 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:29.542692 29005 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:29.542801 29005 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:29.542970 29005 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:29.543116 29005 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260806 09:22:29.588366 29005 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:42121
I20260806 09:22:29.588392 29189 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:42121 every 8 connection(s)
I20260806 09:22:29.589808 29005 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:22:29.592235 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 29005
I20260806 09:22:29.592432 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:22:29.592983 16730 external_mini_cluster.cc:1428] Setting key 70e20bb60c4d234ca68cbfecd05fee93
Aug 06 09:22:29 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008149, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:22:29.621289 29196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:29.595450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:48531 (local address 127.16.86.190:41485)
0806 09:22:29.599553 (+ 4103us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:29.599558 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:29.600351 (+ 793us) server_negotiation.cc:407] Connection header received
0806 09:22:29.602686 (+ 2335us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:29.602691 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:29.602825 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:29.602955 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:29.604635 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:29.605434 (+ 799us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:29.606968 (+ 1534us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:29.607420 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:29.607562 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:29.612929 (+ 5367us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:29.612934 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:29.612937 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:29.612972 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:29.616242 (+ 3270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:29.617722 (+ 1480us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:29.617727 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:29.617733 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:29.617907 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:29.618637 (+ 730us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:29.618641 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:29.618644 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:29.618953 (+ 309us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:29.619185 (+ 232us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:29.620687 (+ 1502us) server_negotiation.cc:299] Negotiation successful
0806 09:22:29.620980 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3990,"thread_start_us":3869,"threads_started":1}
I20260806 09:22:29.623669 29194 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:29.594876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:48531)
0806 09:22:29.600176 (+ 5300us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:29.600231 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:29.602339 (+ 2108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:29.603265 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:29.603273 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:29.603848 (+ 575us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:29.604446 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:29.604463 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:29.605590 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:29.605601 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:29.606819 (+ 1218us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:29.606832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:29.607686 (+ 854us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:29.607694 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:29.607945 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:29.609016 (+ 1071us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:29.609044 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:29.612632 (+ 3588us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:29.616446 (+ 3814us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:29.616463 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:29.616492 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:29.617480 (+ 988us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:29.618039 (+ 559us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:29.618048 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:29.618056 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:29.618424 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:29.619177 (+ 753us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:29.619190 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:29.620011 (+ 821us) client_negotiation.cc:769] Sending connection context
0806 09:22:29.620451 (+ 440us) client_negotiation.cc:240] Negotiation successful
0806 09:22:29.621420 (+ 969us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4957,"thread_start_us":4730,"threads_started":1}
I20260806 09:22:29.625463 29192 heartbeater.cc:347] Connected to a master server at 127.16.86.190:41485
I20260806 09:22:29.625910 29192 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:29.626750 29192 heartbeater.cc:511] Master 127.16.86.190:41485 requested a full tablet report, sending...
I20260806 09:22:29.628652 28532 ts_manager.cc:194] Registered new tserver with Master: e64dc4e536104f17ab81d8df05e76e14 (127.16.86.131:42121)
I20260806 09:22:29.630376 28532 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:48531
I20260806 09:22:29.636242 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab.
Entry for principal oryx/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab.
WARNING: no policy specified for oryx/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab.
Entry for principal oryx/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab.
WARNING: no policy specified for oryx/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab.
Entry for principal oryx/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab.
I20260806 09:22:29.867867 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 28622
Aug 06 09:22:29 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:22:29.887286 29210 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:29.869532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42924 (local address 127.16.86.129:44991)
0806 09:22:29.870503 (+ 971us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:29.870514 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:29.870843 (+ 329us) server_negotiation.cc:407] Connection header received
0806 09:22:29.871094 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:29.871104 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:29.871430 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:29.871609 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:29.873228 (+ 1619us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:29.874363 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:29.875773 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:29.876425 (+ 652us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:29.876533 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:29.880885 (+ 4352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:29.880899 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:29.880954 (+ 55us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:29.880995 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:29.884379 (+ 3384us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:29.885063 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:29.885070 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:29.885073 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:29.885215 (+ 142us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:29.885557 (+ 342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:29.885560 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:29.885562 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:29.885960 (+ 398us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:29.886223 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:29.886501 (+ 278us) server_negotiation.cc:299] Negotiation successful
0806 09:22:29.886776 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":442,"threads_started":1}
I20260806 09:22:29.888890 28793 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:29.982743 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 28622
I20260806 09:22:30.013306 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 28807
Aug 06 09:22:30 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:22:30.034612 28996 heartbeater.cc:503] Master 127.16.86.190:41485 was elected leader, sending a full tablet report...
I20260806 09:22:30.037387 29215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:30.014601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48578 (local address 127.16.86.130:35661)
0806 09:22:30.015560 (+ 959us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:30.015576 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:30.015623 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:22:30.015739 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:30.015751 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:30.016172 (+ 421us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:30.016510 (+ 338us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:30.018178 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:30.020607 (+ 2429us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:30.022339 (+ 1732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:30.023488 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:30.023705 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:30.029593 (+ 5888us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:30.029615 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:30.029639 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:30.029686 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:30.033145 (+ 3459us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:30.034066 (+ 921us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:30.034074 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:30.034076 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:30.034240 (+ 164us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:30.034729 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:30.034735 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:30.034740 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:30.035283 (+ 543us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:30.035646 (+ 363us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:30.036288 (+ 642us) server_negotiation.cc:299] Negotiation successful
0806 09:22:30.036657 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":713,"thread_start_us":495,"threads_started":1}
I20260806 09:22:30.039161 28990 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:30.139077 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 28807
I20260806 09:22:30.164165 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 29005
Aug 06 09:22:30 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008146, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:22:30.184487 29219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:30.165353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53710 (local address 127.16.86.131:42121)
0806 09:22:30.166166 (+ 813us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:30.166176 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:30.166215 (+ 39us) server_negotiation.cc:407] Connection header received
0806 09:22:30.166308 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:30.166321 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:30.166620 (+ 299us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:30.166815 (+ 195us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:30.168259 (+ 1444us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:30.169455 (+ 1196us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:30.170906 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:30.171756 (+ 850us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:30.171936 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:30.177123 (+ 5187us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:30.177139 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:30.177162 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:30.177213 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:30.181009 (+ 3796us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:30.181850 (+ 841us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:30.181899 (+ 49us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:30.181901 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:30.182068 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:30.182442 (+ 374us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:30.182446 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:30.182448 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:30.182911 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:30.183321 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:30.183662 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 09:22:30.184023 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":607,"thread_start_us":403,"threads_started":1}
I20260806 09:22:30.186178 29181 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:30.286180 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 29005
I20260806 09:22:30.318665 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 28486
I20260806 09:22:30.320360 28579 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:30.452456 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 28486
I20260806 09:22:30.482900 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41485
--webserver_interface=127.16.86.190
--webserver_port=37095
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:41485
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:30.840548 29225 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:30.840946 29225 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:30.841135 29225 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:30.849575 29225 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:22:30.849716 29225 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:30.849813 29225 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:30.849929 29225 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:22:30.850008 29225 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:22:30.861138 29225 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:41485
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41485
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=37095
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:30.862542 29225 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:30.863986 29225 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:30.877933 29230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:30.877852 29238 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:30.878861 29232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:30.880983 29225 server_base.cc:1034] running on GCE node
I20260806 09:22:30.881834 29225 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:30.883486 29225 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:30.884786 29225 hybrid_clock.cc:648] HybridClock initialized: now 1786008150884728 us; error 90 us; skew 500 ppm
I20260806 09:22:30.885205 29225 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:30 dist-test-slave-h3ft krb5kdc[28471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008150, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:30.889109 29225 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:22:30.891113 29225 webserver.cc:460] Webserver started at http://127.16.86.190:37095/ using document root <none> and password file <none>
I20260806 09:22:30.891917 29225 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:30.892122 29225 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:30.898495 29225 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.002s
I20260806 09:22:30.902269 29247 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:30.904129 29225 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.002s
I20260806 09:22:30.904393 29225 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "4ced673db7f2454a9cdabd60cdafddc5"
format_stamp: "Formatted at 2026-08-06 09:22:27 on dist-test-slave-h3ft"
server_key: "17afbfa9e9315af04ec4f9652f132b0c"
server_key_iv: "1fe80e7e6bd4251c2a7ab59820efc877"
server_key_version: "encryptionkey@0"
I20260806 09:22:30.905567 29225 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:30.927152 29225 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:30.932307 29225 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:30.932677 29225 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:30.957079 29225 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:41485
I20260806 09:22:30.957191 29328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:41485 every 8 connection(s)
I20260806 09:22:30.958907 29225 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:22:30.968230 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 29225
I20260806 09:22:30.969146 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:44991
--local_ip_for_outbound_sockets=127.16.86.129
--tserver_master_addrs=127.16.86.190:41485
--webserver_port=42873
--webserver_interface=127.16.86.129
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.86.129
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:22:30.969211 29329 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:22:30.974074 29329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5: Bootstrap starting.
I20260806 09:22:31.006712 29329 log.cc:824] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:31.021010 29329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:22:31.021657 29329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5: Bootstrap complete.
I20260806 09:22:31.031421 29329 raft_consensus.cc:348] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced673db7f2454a9cdabd60cdafddc5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41485 } }
I20260806 09:22:31.032328 29329 raft_consensus.cc:729] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4ced673db7f2454a9cdabd60cdafddc5, State: Initialized, Role: FOLLOWER
I20260806 09:22:31.033041 29329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced673db7f2454a9cdabd60cdafddc5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41485 } }
I20260806 09:22:31.033316 29329 raft_consensus.cc:388] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:22:31.033458 29329 raft_consensus.cc:482] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:22:31.033634 29329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:22:31.037685 29329 raft_consensus.cc:504] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced673db7f2454a9cdabd60cdafddc5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41485 } }
I20260806 09:22:31.038224 29329 leader_election.cc:302] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4ced673db7f2454a9cdabd60cdafddc5; no voters:
I20260806 09:22:31.038965 29329 leader_election.cc:288] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 09:22:31.039356 29333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:22:31.040584 29333 raft_consensus.cc:686] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [term 2 LEADER]: Becoming Leader. State: Replica: 4ced673db7f2454a9cdabd60cdafddc5, State: Running, Role: LEADER
I20260806 09:22:31.041538 29329 sys_catalog.cc:564] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:22:31.041354 29333 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced673db7f2454a9cdabd60cdafddc5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41485 } }
I20260806 09:22:31.044582 29335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4ced673db7f2454a9cdabd60cdafddc5. Latest consensus state: current_term: 2 leader_uuid: "4ced673db7f2454a9cdabd60cdafddc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced673db7f2454a9cdabd60cdafddc5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41485 } } }
I20260806 09:22:31.045034 29335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:31.045644 29334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "4ced673db7f2454a9cdabd60cdafddc5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4ced673db7f2454a9cdabd60cdafddc5" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41485 } } }
I20260806 09:22:31.046000 29334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:31.053656 29340 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:22:31.062676 29340 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:22:31.064353 29340 catalog_manager.cc:1259] Loaded cluster ID: 09bf2af412574d3dab7cce71046078cc
I20260806 09:22:31.064479 29340 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:22:31.069239 29340 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:22:31.072199 29340 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 4ced673db7f2454a9cdabd60cdafddc5: Loaded TSK: 0
I20260806 09:22:31.074048 29340 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:22:31.426506 29331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:31.426921 29331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:31.427137 29331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:31.435343 29331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:31.435518 29331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:31.435731 29331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:22:31.446949 29331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.129.keytab
--principal=oryx/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:44991
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=42873
--tserver_master_addrs=127.16.86.190:41485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:31.448786 29331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:31.450485 29331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:31.467043 29362 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:31.467133 29361 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:31.467077 29366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:31.473099 29331 server_base.cc:1034] running on GCE node
I20260806 09:22:31.473994 29331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:31.475214 29331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:31.476493 29331 hybrid_clock.cc:648] HybridClock initialized: now 1786008151476415 us; error 79 us; skew 500 ppm
I20260806 09:22:31.476950 29331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:31 dist-test-slave-h3ft krb5kdc[28471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008151, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:31.480799 29331 init.cc:377] Logged in from keytab as oryx/127.16.86.129@KRBTEST.COM (short username oryx)
I20260806 09:22:31.482782 29331 webserver.cc:460] Webserver started at http://127.16.86.129:42873/ using document root <none> and password file <none>
I20260806 09:22:31.483743 29331 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:31.483887 29331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:31.490806 29331 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 09:22:31.494364 29378 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:31.496239 29331 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:22:31.496433 29331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "307c4ad1acc747ad8b4d29acfb6b6b39"
format_stamp: "Formatted at 2026-08-06 09:22:28 on dist-test-slave-h3ft"
server_key: "a9ebcdafb0f6f51606ee9c506f12b961"
server_key_iv: "1f0bf54f5663ac35e27c88d154349497"
server_key_version: "encryptionkey@0"
I20260806 09:22:31.497639 29331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:31.512524 29331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:31.517230 29331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:31.517520 29331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:31.518673 29331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:31.520299 29331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:31.520401 29331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:31.520507 29331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:31.520571 29331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:31.563309 29331 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:44991
I20260806 09:22:31.563447 29535 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:44991 every 8 connection(s)
I20260806 09:22:31.565019 29331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:22:31.572595 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 29331
I20260806 09:22:31.573518 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:35661
--local_ip_for_outbound_sockets=127.16.86.130
--tserver_master_addrs=127.16.86.190:41485
--webserver_port=41479
--webserver_interface=127.16.86.130
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.86.130
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:22:31 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:31 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 09:22:32.015906 29541 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:32.016376 29541 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:32.016573 29541 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:32.025156 29541 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:32.025352 29541 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:32.025581 29541 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:22:32.037278 29541 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.130.keytab
--principal=oryx/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:35661
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=41479
--tserver_master_addrs=127.16.86.190:41485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:32.039258 29541 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:32.040964 29541 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:32.058621 29548 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:32.059423 29549 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:32.059700 29551 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:32.060921 29541 server_base.cc:1034] running on GCE node
I20260806 09:22:32.062569 29541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:32.064821 29541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:32.066517 29541 hybrid_clock.cc:648] HybridClock initialized: now 1786008152066400 us; error 82 us; skew 500 ppm
I20260806 09:22:32.067265 29541 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:32 dist-test-slave-h3ft krb5kdc[28471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008152, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:32.073939 29541 init.cc:377] Logged in from keytab as oryx/127.16.86.130@KRBTEST.COM (short username oryx)
I20260806 09:22:32.076429 29541 webserver.cc:460] Webserver started at http://127.16.86.130:41479/ using document root <none> and password file <none>
I20260806 09:22:32.077430 29541 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:32.077632 29541 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:32.085870 29541 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260806 09:22:32.090590 29560 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:32.093194 29541 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 09:22:32.093504 29541 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "9597eea916dd47ccad033ae3931d4c96"
format_stamp: "Formatted at 2026-08-06 09:22:28 on dist-test-slave-h3ft"
server_key: "fcca3276b3335fc7fbd44068b062dfc3"
server_key_iv: "1394df0576961562f5257a723843ad6a"
server_key_version: "encryptionkey@0"
I20260806 09:22:32.094813 29541 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:32.117127 29541 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:32.122876 29541 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:32.123239 29541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:32.124789 29541 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:32.126725 29541 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:32.126869 29541 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:32.127071 29541 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:32.127182 29541 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 09:22:32.180747 29541 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:35661
I20260806 09:22:32.180774 29728 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:35661 every 8 connection(s)
I20260806 09:22:32.182634 29541 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:22:32.186437 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 29541
I20260806 09:22:32.187377 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:42121
--local_ip_for_outbound_sockets=127.16.86.131
--tserver_master_addrs=127.16.86.190:41485
--webserver_port=39893
--webserver_interface=127.16.86.131
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44173
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.16.86.131
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:22:32 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:32 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
W20260806 09:22:32.630347 29735 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:32.630734 29735 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:32.630923 29735 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:32.638589 29735 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:32.638762 29735 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:32.638958 29735 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:22:32.649598 29735 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44173
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/krb5kdc/oryx_127.16.86.131.keytab
--principal=oryx/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:42121
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=39893
--tserver_master_addrs=127.16.86.190:41485
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:32.651284 29735 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:32.653466 29735 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:32.673466 29749 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:32.673661 29748 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:32.682703 29751 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:32.683287 29735 server_base.cc:1034] running on GCE node
I20260806 09:22:32.684083 29735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:32.685446 29735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:32.686739 29735 hybrid_clock.cc:648] HybridClock initialized: now 1786008152686667 us; error 65 us; skew 500 ppm
I20260806 09:22:32.687176 29735 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:32 dist-test-slave-h3ft krb5kdc[28471](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008152, etypes {rep=17 tkt=17 ses=17}, oryx/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:32.695219 29735 init.cc:377] Logged in from keytab as oryx/127.16.86.131@KRBTEST.COM (short username oryx)
I20260806 09:22:32.699524 29735 webserver.cc:460] Webserver started at http://127.16.86.131:39893/ using document root <none> and password file <none>
I20260806 09:22:32.700558 29735 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:32.700724 29735 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:32.707324 29735 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:22:32.710603 29760 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:32.712237 29735 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260806 09:22:32.712433 29735 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "e64dc4e536104f17ab81d8df05e76e14"
format_stamp: "Formatted at 2026-08-06 09:22:29 on dist-test-slave-h3ft"
server_key: "5ac8219c266709668ca695c6fa75c4b9"
server_key_iv: "8bec0b0dc8c9fb7d8497160b23d6e531"
server_key_version: "encryptionkey@0"
I20260806 09:22:32.713475 29735 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:32.739439 29735 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:32.744451 29735 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:32.744762 29735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:32.746021 29735 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:32.747784 29735 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:32.747916 29735 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:32.748034 29735 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:32.748143 29735 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:32.809679 29735 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:42121
I20260806 09:22:32.809696 29921 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:42121 every 8 connection(s)
I20260806 09:22:32.811582 29735 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:22:32.817982 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 29735
Aug 06 09:22:32 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:32 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:32.880435 29928 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:32.823441 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57106 (local address 127.16.86.190:41485)
0806 09:22:32.835949 (+ 12508us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:32.835962 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:32.836011 (+ 49us) server_negotiation.cc:407] Connection header received
0806 09:22:32.836136 (+ 125us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:32.836145 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:32.836304 (+ 159us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:32.836558 (+ 254us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:32.839031 (+ 2473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:32.841187 (+ 2156us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:32.845771 (+ 4584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:32.846959 (+ 1188us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:32.847248 (+ 289us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:32.866153 (+ 18905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:32.866177 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:32.866220 (+ 43us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:32.866319 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:32.871971 (+ 5652us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:32.873741 (+ 1770us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:32.873761 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:32.873775 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:32.874161 (+ 386us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:32.874937 (+ 776us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:32.874944 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:32.874948 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:32.875961 (+ 1013us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:32.878066 (+ 2105us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:32.878464 (+ 398us) server_negotiation.cc:299] Negotiation successful
0806 09:22:32.879127 (+ 663us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":111,"server-negotiator.queue_time_us":12201,"thread_start_us":11904,"threads_started":1}
W20260806 09:22:33.003127 29540 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.423s user 0.004s sys 0.013s
W20260806 09:22:33.005223 29539 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:31.569173 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:33909 (local address 127.16.86.190:41485)
0806 09:22:31.570187 (+ 1014us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:31.570206 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:31.570578 (+ 372us) server_negotiation.cc:407] Connection header received
0806 09:22:31.573028 (+ 2450us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:31.573040 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:31.573401 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:31.573769 (+ 368us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:31.575421 (+ 1652us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:31.576779 (+ 1358us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:31.578337 (+ 1558us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:31.579047 (+ 710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:31.579281 (+ 234us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:33.004742 (+1425461us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:33909: BlockingRecv error: recv got EOF from 127.16.86.129:33909 (error 108)
Metrics: {"server-negotiator.queue_time_us":691,"thread_start_us":379,"threads_started":1}
I20260806 09:22:33.007292 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:31.569344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:33909)
0806 09:22:31.570363 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:31.570420 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:31.572423 (+ 2003us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:31.573927 (+ 1504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:31.573937 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:31.574597 (+ 660us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:31.575225 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:31.575245 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:31.576953 (+ 1708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:31.576957 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:31.578190 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:31.578203 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:31.579319 (+ 1116us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:31.579326 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:31.579597 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:31.580476 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:31.580500 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:33.006061 (+1425561us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":439,"threads_started":1}
W20260806 09:22:33.008395 29536 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:41485 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.86.190:41485: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Aug 06 09:22:33 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:33 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:33.022032 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:33.009397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:57925)
0806 09:22:33.009802 (+ 405us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:33.009827 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:33.010070 (+ 243us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:33.010491 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:33.010497 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:33.011167 (+ 670us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:33.011380 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:33.011392 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.012679 (+ 1287us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.012686 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:33.013825 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:33.013840 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.014687 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.014694 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:33.014907 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:33.016155 (+ 1248us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:33.016187 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:33.021624 (+ 5437us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":247}
W20260806 09:22:33.022200 29539 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:33.009575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:57925 (local address 127.16.86.190:41485)
0806 09:22:33.009796 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:33.009804 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:33.009944 (+ 140us) server_negotiation.cc:407] Connection header received
0806 09:22:33.010230 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:33.010236 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:33.010323 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:33.010459 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:33.011543 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.012491 (+ 948us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.013997 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.014474 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.014642 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:33.021888 (+ 7246us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:57925: BlockingRecv error: recv got EOF from 127.16.86.129:57925 (error 108)
Metrics: {"server-negotiator.queue_time_us":74}
Aug 06 09:22:33 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:33 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:33.034968 29540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:33.023285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:37149)
0806 09:22:33.023652 (+ 367us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:33.023669 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:33.023832 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:33.024237 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:33.024245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:33.024616 (+ 371us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:33.024813 (+ 197us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:33.024823 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.026014 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.026028 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:33.027592 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:33.027608 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.028764 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.028770 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:33.029026 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:33.029914 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:33.029949 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:33.034584 (+ 4635us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":269}
W20260806 09:22:33.035493 29536 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:22:33.035657 29539 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:33.023347 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:37149 (local address 127.16.86.190:41485)
0806 09:22:33.023535 (+ 188us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:33.023540 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:33.023752 (+ 212us) server_negotiation.cc:407] Connection header received
0806 09:22:33.023973 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:33.023979 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:33.024077 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:33.024190 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:33.024980 (+ 790us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.025846 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.027752 (+ 1906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.028361 (+ 609us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.028513 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:33.034886 (+ 6373us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:37149: BlockingRecv error: recv got EOF from 127.16.86.129:37149 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260806 09:22:33.692909 29734 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.472s user 0.004s sys 0.020s
I20260806 09:22:33.694656 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:32.187866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:46325)
0806 09:22:32.203016 (+ 15150us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:32.203112 (+ 96us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:32.207140 (+ 4028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:32.208495 (+ 1355us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:32.208514 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:32.209914 (+ 1400us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:32.211323 (+ 1409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:32.211360 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:32.213326 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:32.213336 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:32.216965 (+ 3629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:32.216996 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:32.218565 (+ 1569us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:32.218579 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:32.219131 (+ 552us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:32.220545 (+ 1414us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:32.220619 (+ 74us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:33.693726 (+1473107us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":12970,"thread_start_us":13145,"threads_started":1}
W20260806 09:22:33.694885 29732 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:32.187099 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:46325 (local address 127.16.86.190:41485)
0806 09:22:32.203561 (+ 16462us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:32.203572 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:32.203612 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:22:32.207683 (+ 4071us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:32.207695 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:32.207836 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:32.208025 (+ 189us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:32.211713 (+ 3688us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:32.213069 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:32.217277 (+ 4208us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:32.218132 (+ 855us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:32.218328 (+ 196us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:33.694303 (+1475975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:46325: BlockingRecv error: recv got EOF from 127.16.86.130:46325 (error 108)
Metrics: {"server-negotiator.queue_time_us":16243,"thread_start_us":16509,"threads_started":1}
W20260806 09:22:33.696452 29729 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:41485 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.86.190:41485: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Aug 06 09:22:33 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:33 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:33.708624 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:33.697590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:46011)
0806 09:22:33.697822 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:33.697870 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:33.698010 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:33.698367 (+ 357us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:33.698371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:33.698823 (+ 452us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:33.698996 (+ 173us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:33.699066 (+ 70us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.700241 (+ 1175us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.700246 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:33.701422 (+ 1176us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:33.701454 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.702208 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.702215 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:33.702393 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:33.703528 (+ 1135us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:33.703546 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:33.708309 (+ 4763us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 09:22:33.708807 29732 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:33.697757 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:46011 (local address 127.16.86.190:41485)
0806 09:22:33.698018 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:33.698025 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:33.698050 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:22:33.698122 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:33.698128 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:33.698204 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:33.698337 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:33.699202 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.700090 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.701590 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.702010 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.702154 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:33.708548 (+ 6394us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:46011: BlockingRecv error: recv got EOF from 127.16.86.130:46011 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
Aug 06 09:22:33 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:33 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:33.724467 29734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:33.709447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:38263)
0806 09:22:33.709817 (+ 370us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:33.709857 (+ 40us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:33.710148 (+ 291us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:33.710623 (+ 475us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:33.710632 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:33.711451 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:33.711893 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:33.711906 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.713093 (+ 1187us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.713102 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:33.715683 (+ 2581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:33.715709 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.716598 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.716608 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:33.717081 (+ 473us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:33.718432 (+ 1351us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:33.718472 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:33.724119 (+ 5647us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":105}
W20260806 09:22:33.724725 29732 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:33.709643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:38263 (local address 127.16.86.190:41485)
0806 09:22:33.709818 (+ 175us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:33.709822 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:33.709981 (+ 159us) server_negotiation.cc:407] Connection header received
0806 09:22:33.710328 (+ 347us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:33.710332 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:33.710406 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:33.710512 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:33.712124 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.712900 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.715903 (+ 3003us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:33.716366 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:33.716481 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:33.724404 (+ 7923us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:38263: BlockingRecv error: recv got EOF from 127.16.86.130:38263 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
W20260806 09:22:33.724910 29729 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 09:22:34 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:34 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:34.047659 29930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:34.036300 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:46885)
0806 09:22:34.037210 (+ 910us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:34.037240 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:34.037455 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:34.037926 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:34.037931 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:34.038566 (+ 635us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:34.038783 (+ 217us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:34.038801 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.039896 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.039901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:34.040876 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:34.040888 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.041641 (+ 753us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.041647 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:34.041814 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:34.042842 (+ 1028us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:34.042862 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:34.047246 (+ 4384us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":781,"thread_start_us":824,"threads_started":1}
W20260806 09:22:34.047808 29732 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:34.036336 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:46885 (local address 127.16.86.190:41485)
0806 09:22:34.036738 (+ 402us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:34.036748 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:34.037327 (+ 579us) server_negotiation.cc:407] Connection header received
0806 09:22:34.037663 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:34.037669 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:34.037771 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:34.037880 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:34.038995 (+ 1115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.039747 (+ 752us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.041024 (+ 1277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.041439 (+ 415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.041583 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:34.047485 (+ 5902us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:46885: BlockingRecv error: recv got EOF from 127.16.86.129:46885 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260806 09:22:34.165277 29924 sasl_common.cc:323] Time spent calling sasl_client_start(): real 1.325s user 0.001s sys 0.025s
W20260806 09:22:34.166932 29925 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:32.819057 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:51307 (local address 127.16.86.190:41485)
0806 09:22:32.823559 (+ 4502us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:32.823568 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:32.828099 (+ 4531us) server_negotiation.cc:407] Connection header received
0806 09:22:32.828214 (+ 115us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:32.828223 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:32.828366 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:32.828502 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:32.832400 (+ 3898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:32.834209 (+ 1809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:32.836730 (+ 2521us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:32.837752 (+ 1022us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:32.837910 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:34.166584 (+1328674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:51307: BlockingRecv error: recv got EOF from 127.16.86.131:51307 (error 108)
Metrics: {"mutex_wait_us":54,"server-negotiator.queue_time_us":4311,"thread_start_us":4182,"threads_started":1}
I20260806 09:22:34.167204 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:32.816499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:51307)
0806 09:22:32.824197 (+ 7698us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:32.824250 (+ 53us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:32.827549 (+ 3299us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:32.829429 (+ 1880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:32.829449 (+ 20us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:32.830733 (+ 1284us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:32.831999 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:32.832039 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:32.834556 (+ 2517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:32.834565 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:32.836437 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:32.836454 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:32.838369 (+ 1915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:32.838379 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:32.838760 (+ 381us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:32.839794 (+ 1034us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:32.839845 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:34.166158 (+1326313us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":7350,"thread_start_us":7065,"threads_started":1}
W20260806 09:22:34.168524 29922 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:41485 (0 consecutive failures): Not authorized: Failed to ping master at 127.16.86.190:41485: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Aug 06 09:22:34 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:34 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:34.179390 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:34.169434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:52235)
0806 09:22:34.169703 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:34.169727 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:34.169855 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:34.170326 (+ 471us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:34.170333 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:34.170872 (+ 539us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:34.171140 (+ 268us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:34.171152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.172305 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.172309 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:34.173433 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:34.173448 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.174239 (+ 791us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.174244 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:34.174384 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:34.175278 (+ 894us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:34.175296 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:34.179157 (+ 3861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":125}
W20260806 09:22:34.179774 29925 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:34.169579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:52235 (local address 127.16.86.190:41485)
0806 09:22:34.169887 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:34.169892 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:34.169914 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:22:34.169998 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:34.170003 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:34.170104 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:34.170237 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:34.171297 (+ 1060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.172142 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.173592 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.174053 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.174163 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:34.179463 (+ 5300us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:52235: BlockingRecv error: recv got EOF from 127.16.86.131:52235 (error 108)
Metrics: {"server-negotiator.queue_time_us":170}
Aug 06 09:22:34 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:34 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:34.190760 29924 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:34.180552 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:33415)
0806 09:22:34.180768 (+ 216us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:34.180795 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:34.180905 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:34.181382 (+ 477us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:34.181387 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:34.181820 (+ 433us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:34.182027 (+ 207us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:34.182040 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.183615 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.183623 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:34.184715 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:34.184730 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.185541 (+ 811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.185547 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:34.185766 (+ 219us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:34.186497 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:34.186517 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:34.190394 (+ 3877us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":81}
W20260806 09:22:34.191020 29925 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:34.180658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:33415 (local address 127.16.86.190:41485)
0806 09:22:34.180858 (+ 200us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:34.180862 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:34.180882 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:22:34.181089 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:34.181094 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:34.181201 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:34.181323 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:34.182190 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.183221 (+ 1031us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.184877 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.185374 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.185520 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:34.190664 (+ 5144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:33415: BlockingRecv error: recv got EOF from 127.16.86.131:33415 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20260806 09:22:34.191229 29922 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Aug 06 09:22:34 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:34 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:34.742622 29931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:34.725691 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:55631)
0806 09:22:34.726386 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:34.726405 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:34.726558 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:34.726981 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:34.726985 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:34.727605 (+ 620us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:34.727794 (+ 189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:34.727848 (+ 54us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.728958 (+ 1110us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.728962 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:34.730061 (+ 1099us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:34.730074 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.730988 (+ 914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.730994 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:34.731225 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:34.737727 (+ 6502us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:34.737752 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:34.742298 (+ 4546us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":412,"threads_started":1}
W20260806 09:22:34.743047 29932 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:34.725971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:55631 (local address 127.16.86.190:41485)
0806 09:22:34.726504 (+ 533us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:34.726509 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:34.726529 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:22:34.726703 (+ 174us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:34.726707 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:34.726796 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:34.726908 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:34.727978 (+ 1070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.728788 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.730217 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:34.730667 (+ 450us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:34.730775 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:34.742636 (+ 11861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:55631: BlockingRecv error: recv got EOF from 127.16.86.130:55631 (error 108)
Metrics: {"server-negotiator.queue_time_us":429,"thread_start_us":311,"threads_started":1}
Aug 06 09:22:35 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:35 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:35.060705 29935 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:35.048885 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:58417)
0806 09:22:35.049567 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:35.049597 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:35.049779 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:35.050204 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:35.050209 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:35.050767 (+ 558us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:35.051097 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:35.051112 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:35.052378 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:35.052385 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:35.054053 (+ 1668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:35.054076 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:35.055089 (+ 1013us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:35.055097 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:35.055298 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:35.056122 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:35.056144 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:35.060391 (+ 4247us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":395,"threads_started":1}
W20260806 09:22:35.060814 29932 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:35.049212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:58417 (local address 127.16.86.190:41485)
0806 09:22:35.049478 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:35.049485 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:35.049717 (+ 232us) server_negotiation.cc:407] Connection header received
0806 09:22:35.049910 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:35.049913 (+ 3us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:35.049986 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:35.050097 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:35.051264 (+ 1167us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:35.052163 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:35.054235 (+ 2072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:35.054776 (+ 541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:35.054929 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:35.060608 (+ 5679us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:58417: BlockingRecv error: recv got EOF from 127.16.86.129:58417 (error 108)
Metrics: {"server-negotiator.queue_time_us":110}
Aug 06 09:22:35 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:35 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:35.204967 29937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:35.192161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:34181)
0806 09:22:35.193011 (+ 850us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:35.193032 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:35.193254 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:35.193887 (+ 633us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:35.193891 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:35.194542 (+ 651us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:35.194821 (+ 279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:35.194835 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:35.196329 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:35.196335 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:35.197535 (+ 1200us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:35.197554 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:35.198470 (+ 916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:35.198476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:35.198631 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:35.199552 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:35.199576 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:35.204666 (+ 5090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":733,"thread_start_us":512,"threads_started":1}
W20260806 09:22:35.205469 29932 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:35.192379 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:34181 (local address 127.16.86.190:41485)
0806 09:22:35.192745 (+ 366us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:35.192755 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:35.193166 (+ 411us) server_negotiation.cc:407] Connection header received
0806 09:22:35.193512 (+ 346us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:35.193518 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:35.193672 (+ 154us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:35.193839 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:35.195041 (+ 1202us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:35.196070 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:35.197734 (+ 1664us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:35.198189 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:35.198308 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:35.204979 (+ 6671us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:34181: BlockingRecv error: recv got EOF from 127.16.86.131:34181 (error 108)
Metrics: {"server-negotiator.queue_time_us":128}
Aug 06 09:22:35 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:35 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:35.756147 29938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:35.744075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:38061)
0806 09:22:35.745005 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:35.745025 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:35.745176 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:35.745593 (+ 417us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:35.745596 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:35.746085 (+ 489us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:35.746305 (+ 220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:35.746314 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:35.747507 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:35.747512 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:35.748605 (+ 1093us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:35.748619 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:35.749404 (+ 785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:35.749409 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:35.749554 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:35.750301 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:35.750320 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:35.755816 (+ 5496us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":776,"thread_start_us":641,"threads_started":1}
W20260806 09:22:35.756309 29939 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:35.744103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:38061 (local address 127.16.86.190:41485)
0806 09:22:35.744706 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:35.744712 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:35.745166 (+ 454us) server_negotiation.cc:407] Connection header received
0806 09:22:35.745310 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:35.745315 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:35.745404 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:35.745527 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:35.746489 (+ 962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:35.747334 (+ 845us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:35.748765 (+ 1431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:35.749228 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:35.749336 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:35.756022 (+ 6686us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:38061: BlockingRecv error: recv got EOF from 127.16.86.130:38061 (error 108)
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":327,"threads_started":1}
Aug 06 09:22:36 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:36 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:36.074692 29941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:36.062103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:42323)
0806 09:22:36.063120 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:36.063146 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:36.063313 (+ 167us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:36.063806 (+ 493us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:36.063810 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:36.064294 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:36.064518 (+ 224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:36.064529 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:36.065697 (+ 1168us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:36.065704 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:36.066756 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:36.066770 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:36.067722 (+ 952us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:36.067727 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:36.067931 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:36.068735 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:36.068760 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:36.074272 (+ 5512us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":863,"thread_start_us":367,"threads_started":1}
W20260806 09:22:36.074918 29939 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:36.062528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:42323 (local address 127.16.86.190:41485)
0806 09:22:36.062811 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:36.062819 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:36.063266 (+ 447us) server_negotiation.cc:407] Connection header received
0806 09:22:36.063508 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:36.063512 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:36.063594 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:36.063708 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:36.064719 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:36.065525 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:36.067041 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:36.067508 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:36.067696 (+ 188us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:36.074623 (+ 6927us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:42323: BlockingRecv error: recv got EOF from 127.16.86.129:42323 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 09:22:36 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:36 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:36.218312 29944 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:36.206155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:46097)
0806 09:22:36.206916 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:36.206935 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:36.207193 (+ 258us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:36.207611 (+ 418us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:36.207615 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:36.208203 (+ 588us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:36.208434 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:36.208444 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:36.209617 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:36.209624 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:36.210896 (+ 1272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:36.210910 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:36.211782 (+ 872us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:36.211788 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:36.212083 (+ 295us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:36.212860 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:36.212881 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:36.217953 (+ 5072us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":634,"thread_start_us":440,"threads_started":1}
W20260806 09:22:36.218767 29939 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:36.206342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:46097 (local address 127.16.86.190:41485)
0806 09:22:36.206654 (+ 312us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:36.206663 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:36.207070 (+ 407us) server_negotiation.cc:407] Connection header received
0806 09:22:36.207345 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:36.207358 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:36.207453 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:36.207580 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:36.208591 (+ 1011us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:36.209449 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:36.211088 (+ 1639us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:36.211587 (+ 499us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:36.211754 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:36.218224 (+ 6470us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:46097: BlockingRecv error: recv got EOF from 127.16.86.131:46097 (error 108)
Metrics: {"server-negotiator.queue_time_us":97}
Aug 06 09:22:36 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:36 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:36.769156 29947 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:36.757649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:41293)
0806 09:22:36.758383 (+ 734us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:36.758405 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:36.758564 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:36.759305 (+ 741us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:36.759309 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:36.759878 (+ 569us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:36.760153 (+ 275us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:36.760171 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:36.761724 (+ 1553us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:36.761730 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:36.762805 (+ 1075us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:36.762819 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:36.763702 (+ 883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:36.763707 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:36.763848 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:36.764646 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:36.764666 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:36.768885 (+ 4219us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":358,"threads_started":1}
W20260806 09:22:36.769805 29949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:36.757822 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:41293 (local address 127.16.86.190:41485)
0806 09:22:36.758818 (+ 996us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:36.758824 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:36.758848 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:22:36.758925 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:36.758929 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:36.759058 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:36.759183 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:36.760397 (+ 1214us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:36.761577 (+ 1180us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:36.763022 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:36.763490 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:36.763615 (+ 125us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:36.769244 (+ 5629us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:41293: BlockingRecv error: recv got EOF from 127.16.86.130:41293 (error 108)
Metrics: {"server-negotiator.queue_time_us":864,"thread_start_us":677,"threads_started":1}
Aug 06 09:22:37 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:37 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:37.088203 29952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:37.076310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:53405)
0806 09:22:37.077064 (+ 754us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:37.077094 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:37.077325 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:37.077769 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:37.077775 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:37.078308 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:37.078517 (+ 209us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:37.078529 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:37.079821 (+ 1292us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:37.079827 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:37.080961 (+ 1134us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:37.080977 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:37.081818 (+ 841us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:37.081825 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:37.082051 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:37.082894 (+ 843us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:37.082924 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:37.087870 (+ 4946us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":641,"thread_start_us":394,"threads_started":1}
W20260806 09:22:37.088800 29949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:37.076553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:53405 (local address 127.16.86.190:41485)
0806 09:22:37.076878 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:37.076886 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:37.077225 (+ 339us) server_negotiation.cc:407] Connection header received
0806 09:22:37.077503 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:37.077507 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:37.077589 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:37.077710 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:37.078686 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:37.079663 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:37.081136 (+ 1473us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:37.081638 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:37.081791 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:37.088192 (+ 6401us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:53405: BlockingRecv error: recv got EOF from 127.16.86.129:53405 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 06 09:22:37 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:37 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:37.230528 29953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:37.219559 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:34365)
0806 09:22:37.220559 (+ 1000us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:37.220579 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:37.220787 (+ 208us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:37.221210 (+ 423us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:37.221215 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:37.221757 (+ 542us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:37.221996 (+ 239us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:37.222006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:37.223149 (+ 1143us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:37.223157 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:37.224140 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:37.224152 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:37.224936 (+ 784us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:37.224942 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:37.225148 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:37.225878 (+ 730us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:37.225944 (+ 66us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:37.230135 (+ 4191us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":406,"threads_started":1}
W20260806 09:22:37.230715 29949 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:37.219811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:34365 (local address 127.16.86.190:41485)
0806 09:22:37.220221 (+ 410us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:37.220229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:37.220695 (+ 466us) server_negotiation.cc:407] Connection header received
0806 09:22:37.220933 (+ 238us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:37.220939 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:37.221053 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:37.221178 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:37.222158 (+ 980us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:37.222953 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:37.224296 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:37.224766 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:37.224931 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:37.230418 (+ 5487us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:34365: BlockingRecv error: recv got EOF from 127.16.86.131:34365 (error 108)
Metrics: {"server-negotiator.queue_time_us":199}
Aug 06 09:22:37 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:37 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:37.787878 29955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:37.770845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:54293)
0806 09:22:37.771890 (+ 1045us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:37.771911 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:37.772047 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:37.772615 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:37.772625 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:37.773290 (+ 665us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:37.773535 (+ 245us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:37.773546 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:37.774913 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:37.774919 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:37.776189 (+ 1270us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:37.776204 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:37.777129 (+ 925us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:37.777134 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:37.777308 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:37.782102 (+ 4794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:37.782132 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:37.787449 (+ 5317us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":872,"thread_start_us":729,"threads_started":1}
W20260806 09:22:37.788077 29956 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:37.770991 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:54293 (local address 127.16.86.190:41485)
0806 09:22:37.772149 (+ 1158us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:37.772156 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:37.772183 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:22:37.772262 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:37.772271 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:37.772370 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:37.772491 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:37.773743 (+ 1252us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:37.774701 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:37.776393 (+ 1692us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:37.776914 (+ 521us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:37.777042 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:37.787735 (+ 10693us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:54293: BlockingRecv error: recv got EOF from 127.16.86.130:54293 (error 108)
Metrics: {"server-negotiator.queue_time_us":634,"thread_start_us":501,"threads_started":1}
Aug 06 09:22:38 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:38 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:38.101411 29957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:38.090043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:47523)
0806 09:22:38.090713 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:38.090746 (+ 33us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:38.090948 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:38.091604 (+ 656us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:38.091607 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:38.092139 (+ 532us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:38.092327 (+ 188us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:38.092337 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:38.093467 (+ 1130us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:38.093472 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:38.094536 (+ 1064us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:38.094550 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:38.095314 (+ 764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:38.095319 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:38.095506 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:38.096254 (+ 748us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:38.096275 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:38.101048 (+ 4773us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":514,"thread_start_us":370,"threads_started":1}
W20260806 09:22:38.101557 29956 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:38.090168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:47523 (local address 127.16.86.190:41485)
0806 09:22:38.090400 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:38.090408 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:38.090861 (+ 453us) server_negotiation.cc:407] Connection header received
0806 09:22:38.091265 (+ 404us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:38.091269 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:38.091347 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:38.091410 (+ 63us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:38.092487 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:38.093308 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:38.094673 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:38.095126 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:38.095242 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:38.101321 (+ 6079us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:47523: BlockingRecv error: recv got EOF from 127.16.86.129:47523 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
Aug 06 09:22:38 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:38 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:38.243908 29958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:38.232041 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:47605)
0806 09:22:38.232765 (+ 724us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:38.232786 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:38.233024 (+ 238us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:38.233592 (+ 568us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:38.233598 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:38.234281 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:38.234529 (+ 248us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:38.234541 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:38.235704 (+ 1163us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:38.235708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:38.236838 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:38.236853 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:38.237653 (+ 800us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:38.237667 (+ 14us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:38.237823 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:38.238552 (+ 729us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:38.238572 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:38.243581 (+ 5009us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":360,"threads_started":1}
W20260806 09:22:38.244427 29956 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:38.232127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:47605 (local address 127.16.86.190:41485)
0806 09:22:38.232437 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:38.232445 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:38.232948 (+ 503us) server_negotiation.cc:407] Connection header received
0806 09:22:38.233219 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:38.233242 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:38.233336 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:38.233459 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:38.234726 (+ 1267us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:38.235547 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:38.236997 (+ 1450us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:38.237480 (+ 483us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:38.237588 (+ 108us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:38.244002 (+ 6414us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:47605: BlockingRecv error: recv got EOF from 127.16.86.131:47605 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Aug 06 09:22:38 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:38 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:38.800170 29960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:38.789255 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:58295)
0806 09:22:38.789947 (+ 692us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:38.789969 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:38.790122 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:38.790750 (+ 628us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:38.790754 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:38.791337 (+ 583us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:38.791531 (+ 194us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:38.791540 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:38.792692 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:38.792696 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:38.793674 (+ 978us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:38.793685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:38.794447 (+ 762us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:38.794452 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:38.794580 (+ 128us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:38.795330 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:38.795348 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:38.799728 (+ 4380us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":511,"thread_start_us":346,"threads_started":1}
W20260806 09:22:38.800387 29961 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:38.789531 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:58295 (local address 127.16.86.190:41485)
0806 09:22:38.790315 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:38.790321 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:38.790342 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:22:38.790409 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:38.790413 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:38.790499 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:38.790618 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:38.791684 (+ 1066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:38.792504 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:38.793831 (+ 1327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:38.794298 (+ 467us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:38.794405 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:38.800007 (+ 5602us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:58295: BlockingRecv error: recv got EOF from 127.16.86.130:58295 (error 108)
Metrics: {"server-negotiator.queue_time_us":664,"thread_start_us":529,"threads_started":1}
Aug 06 09:22:39 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:39 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:39.114123 29963 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:39.102777 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:46203)
0806 09:22:39.103427 (+ 650us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:39.103450 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:39.103601 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:39.104096 (+ 495us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:39.104100 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:39.104569 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:39.104806 (+ 237us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:39.104816 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:39.106127 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:39.106132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:39.107430 (+ 1298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:39.107447 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:39.108386 (+ 939us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:39.108391 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:39.108526 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:39.109438 (+ 912us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:39.109457 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:39.113844 (+ 4387us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":384,"threads_started":1}
W20260806 09:22:39.114665 29961 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:39.103076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:46203 (local address 127.16.86.190:41485)
0806 09:22:39.103302 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:39.103308 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:39.103596 (+ 288us) server_negotiation.cc:407] Connection header received
0806 09:22:39.103775 (+ 179us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:39.103780 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:39.103872 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:39.104022 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:39.104992 (+ 970us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:39.105928 (+ 936us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:39.107638 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:39.108190 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:39.108351 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:39.114271 (+ 5920us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:46203: BlockingRecv error: recv got EOF from 127.16.86.129:46203 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
Aug 06 09:22:39 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:39 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:39.257349 29964 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:39.245230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:41967)
0806 09:22:39.246041 (+ 811us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:39.246070 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:39.246299 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:39.246852 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:39.246858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:39.247615 (+ 757us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:39.247886 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:39.247901 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:39.249271 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:39.249276 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:39.250433 (+ 1157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:39.250446 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:39.251432 (+ 986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:39.251438 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:39.251584 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:39.252374 (+ 790us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:39.252393 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:39.257061 (+ 4668us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":388,"threads_started":1}
W20260806 09:22:39.257704 29961 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:39.245430 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:41967 (local address 127.16.86.190:41485)
0806 09:22:39.245782 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:39.245791 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:39.246263 (+ 472us) server_negotiation.cc:407] Connection header received
0806 09:22:39.246451 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:39.246457 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:39.246559 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:39.246717 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:39.248096 (+ 1379us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:39.249073 (+ 977us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:39.250606 (+ 1533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:39.251224 (+ 618us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:39.251383 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:39.257358 (+ 5975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:41967: BlockingRecv error: recv got EOF from 127.16.86.131:41967 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Aug 06 09:22:39 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:39 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:39.813954 29968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:39.801514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:45433)
0806 09:22:39.803788 (+ 2274us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:39.803814 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:39.803950 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:39.804385 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:39.804388 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:39.804827 (+ 439us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:39.804998 (+ 171us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:39.805006 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:39.806227 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:39.806254 (+ 27us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:39.807273 (+ 1019us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:39.807287 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:39.808152 (+ 865us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:39.808157 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:39.808301 (+ 144us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:39.809066 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:39.809086 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:39.813495 (+ 4409us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":2133,"thread_start_us":2008,"threads_started":1}
W20260806 09:22:39.814497 29965 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:39.801583 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:45433 (local address 127.16.86.190:41485)
0806 09:22:39.803525 (+ 1942us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:39.803532 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:39.803927 (+ 395us) server_negotiation.cc:407] Connection header received
0806 09:22:39.804087 (+ 160us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:39.804092 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:39.804194 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:39.804337 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:39.805174 (+ 837us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:39.806060 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:39.807428 (+ 1368us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:39.807955 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:39.808086 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:39.813852 (+ 5766us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:45433: BlockingRecv error: recv got EOF from 127.16.86.130:45433 (error 108)
Metrics: {"server-negotiator.queue_time_us":1801,"thread_start_us":1652,"threads_started":1}
Aug 06 09:22:40 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:40 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:40.127667 29969 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:40.115495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:34407)
0806 09:22:40.116350 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:40.116376 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:40.116540 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:40.116919 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:40.116923 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:40.117557 (+ 634us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:40.117817 (+ 260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:40.117828 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:40.119118 (+ 1290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:40.119123 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:40.120309 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:40.120324 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:40.121097 (+ 773us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:40.121102 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:40.121243 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:40.122003 (+ 760us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:40.122025 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:40.127302 (+ 5277us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":687,"thread_start_us":424,"threads_started":1}
W20260806 09:22:40.127725 29965 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:40.115575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:34407 (local address 127.16.86.190:41485)
0806 09:22:40.115774 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:40.115779 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:40.116479 (+ 700us) server_negotiation.cc:407] Connection header received
0806 09:22:40.116662 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:40.116666 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:40.116746 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:40.116897 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:40.118001 (+ 1104us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:40.118915 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:40.120485 (+ 1570us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:40.120929 (+ 444us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:40.121038 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:40.127497 (+ 6459us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:34407: BlockingRecv error: recv got EOF from 127.16.86.129:34407 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
Aug 06 09:22:40 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:40 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:40.269420 29970 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:40.258675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:35519)
0806 09:22:40.259297 (+ 622us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:40.259314 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:40.259442 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:40.259842 (+ 400us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:40.259846 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:40.260300 (+ 454us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:40.260484 (+ 184us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:40.260493 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:40.261699 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:40.261703 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:40.262670 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:40.262682 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:40.263552 (+ 870us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:40.263556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:40.263715 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:40.264381 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:40.264398 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:40.269125 (+ 4727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":485,"thread_start_us":372,"threads_started":1}
W20260806 09:22:40.269688 29965 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:40.258811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:35519 (local address 127.16.86.190:41485)
0806 09:22:40.259110 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:40.259116 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:40.259401 (+ 285us) server_negotiation.cc:407] Connection header received
0806 09:22:40.259563 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:40.259567 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:40.259678 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:40.259794 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:40.260649 (+ 855us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:40.261529 (+ 880us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:40.262829 (+ 1300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:40.263384 (+ 555us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:40.263514 (+ 130us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:40.269366 (+ 5852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:35519: BlockingRecv error: recv got EOF from 127.16.86.131:35519 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 06 09:22:40 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:40 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:40.827829 29974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:40.815322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:39065)
0806 09:22:40.816206 (+ 884us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:40.816253 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:40.816424 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:40.816971 (+ 547us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:40.816976 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:40.817606 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:40.817960 (+ 354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:40.817971 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:40.819523 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:40.819528 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:40.820778 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:40.820795 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:40.821675 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:40.821684 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:40.821957 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:40.822781 (+ 824us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:40.822819 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:40.827433 (+ 4614us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":727,"thread_start_us":428,"threads_started":1}
W20260806 09:22:40.827986 29977 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:40.815460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:39065 (local address 127.16.86.190:41485)
0806 09:22:40.816573 (+ 1113us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:40.816579 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:40.816602 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:22:40.816665 (+ 63us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:40.816669 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:40.816755 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:40.816872 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:40.818151 (+ 1279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:40.819344 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:40.820999 (+ 1655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:40.821454 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:40.821611 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:40.827644 (+ 6033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:39065: BlockingRecv error: recv got EOF from 127.16.86.130:39065 (error 108)
Metrics: {"server-negotiator.queue_time_us":963,"thread_start_us":843,"threads_started":1}
Aug 06 09:22:41 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:41 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:41.140763 29978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:41.129194 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:46539)
0806 09:22:41.129840 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:41.129864 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:41.130011 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:41.130475 (+ 464us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:41.130480 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:41.130979 (+ 499us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:41.131263 (+ 284us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:41.131283 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:41.132685 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:41.132691 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:41.133925 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:41.133944 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:41.135042 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:41.135051 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:41.135218 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:41.136038 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:41.136059 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:41.140433 (+ 4374us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":342,"threads_started":1}
W20260806 09:22:41.141010 29977 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:41.129490 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:46539 (local address 127.16.86.190:41485)
0806 09:22:41.129736 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:41.129741 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:41.130004 (+ 263us) server_negotiation.cc:407] Connection header received
0806 09:22:41.130125 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:41.130130 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:41.130226 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:41.130374 (+ 148us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:41.131517 (+ 1143us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:41.132468 (+ 951us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:41.134136 (+ 1668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:41.134750 (+ 614us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:41.134881 (+ 131us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:41.140742 (+ 5861us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:46539: BlockingRecv error: recv got EOF from 127.16.86.129:46539 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
Aug 06 09:22:41 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:41 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:41.282078 29979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:41.270592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:48543)
0806 09:22:41.271561 (+ 969us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:41.271582 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:41.271791 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:41.272241 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:41.272247 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:41.272783 (+ 536us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:41.272996 (+ 213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:41.273006 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:41.274266 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:41.274274 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:41.275780 (+ 1506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:41.275800 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:41.276653 (+ 853us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:41.276660 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:41.276828 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:41.277530 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:41.277556 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:41.281728 (+ 4172us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":843,"thread_start_us":529,"threads_started":1}
W20260806 09:22:41.282245 29977 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:41.271074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:48543 (local address 127.16.86.190:41485)
0806 09:22:41.271397 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:41.271406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:41.271696 (+ 290us) server_negotiation.cc:407] Connection header received
0806 09:22:41.271949 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:41.271955 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:41.272065 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:41.272183 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:41.273171 (+ 988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:41.273987 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:41.275973 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:41.276476 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:41.276630 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:41.281971 (+ 5341us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:48543: BlockingRecv error: recv got EOF from 127.16.86.131:48543 (error 108)
Metrics: {"server-negotiator.queue_time_us":118}
Aug 06 09:22:41 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:41 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:41.839955 29981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:41.829074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:57313)
0806 09:22:41.829699 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:41.829722 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:41.829868 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:41.830351 (+ 483us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:41.830354 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:41.830903 (+ 549us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:41.831185 (+ 282us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:41.831196 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:41.832734 (+ 1538us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:41.832738 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:41.833753 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:41.833766 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:41.834681 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:41.834686 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:41.834815 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:41.835608 (+ 793us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:41.835637 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:41.839677 (+ 4040us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":425,"thread_start_us":289,"threads_started":1}
W20260806 09:22:41.840508 29982 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:41.829300 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:57313 (local address 127.16.86.190:41485)
0806 09:22:41.829969 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:41.829975 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:41.829995 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:22:41.830059 (+ 64us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:41.830063 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:41.830157 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:41.830297 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:41.831389 (+ 1092us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:41.832528 (+ 1139us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:41.833925 (+ 1397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:41.834457 (+ 532us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:41.834578 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:41.840121 (+ 5543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:57313: BlockingRecv error: recv got EOF from 127.16.86.130:57313 (error 108)
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":379,"threads_started":1}
Aug 06 09:22:42 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:42 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:42.153577 29984 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:42.142184 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:44235)
0806 09:22:42.142948 (+ 764us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:42.142973 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:42.143167 (+ 194us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:42.143648 (+ 481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:42.143652 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:42.144156 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:42.144333 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:42.144344 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:42.145503 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:42.145507 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:42.146621 (+ 1114us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:42.146636 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:42.147525 (+ 889us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:42.147530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:42.147706 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:42.148452 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:42.148473 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:42.153150 (+ 4677us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":625,"thread_start_us":458,"threads_started":1}
W20260806 09:22:42.153676 29982 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:42.142244 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:44235 (local address 127.16.86.190:41485)
0806 09:22:42.142476 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:42.142483 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:42.143135 (+ 652us) server_negotiation.cc:407] Connection header received
0806 09:22:42.143323 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:42.143328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:42.143434 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:42.143570 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:42.144485 (+ 915us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:42.145343 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:42.146788 (+ 1445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:42.147303 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:42.147415 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:42.153400 (+ 5985us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:44235: BlockingRecv error: recv got EOF from 127.16.86.129:44235 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
Aug 06 09:22:42 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:42 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:42.294324 29985 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:42.283437 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:36097)
0806 09:22:42.284219 (+ 782us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:42.284239 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:42.284428 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:42.284802 (+ 374us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:42.284807 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:42.285298 (+ 491us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:42.285472 (+ 174us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:42.285481 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:42.286571 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:42.286575 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:42.287586 (+ 1011us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:42.287600 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:42.288515 (+ 915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:42.288545 (+ 30us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:42.288741 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:42.289535 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:42.289555 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:42.294002 (+ 4447us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":422,"threads_started":1}
W20260806 09:22:42.294553 29982 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:42.283627 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:36097 (local address 127.16.86.190:41485)
0806 09:22:42.283953 (+ 326us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:42.283967 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:42.284344 (+ 377us) server_negotiation.cc:407] Connection header received
0806 09:22:42.284561 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:42.284566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:42.284656 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:42.284766 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:42.285635 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:42.286416 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:42.287738 (+ 1322us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:42.288244 (+ 506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:42.288395 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:42.294274 (+ 5879us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:36097: BlockingRecv error: recv got EOF from 127.16.86.131:36097 (error 108)
Metrics: {"server-negotiator.queue_time_us":127}
Aug 06 09:22:42 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:42 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:42.860034 29986 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:42.841320 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:53413)
0806 09:22:42.842035 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:42.842065 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:42.842251 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:42.842816 (+ 565us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:42.842823 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:42.843622 (+ 799us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:42.843898 (+ 276us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:42.843914 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:42.845641 (+ 1727us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:42.845648 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:42.847375 (+ 1727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:42.847389 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:42.848728 (+ 1339us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:42.848735 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:42.848943 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:42.854661 (+ 5718us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:42.854688 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:42.859658 (+ 4970us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":512,"thread_start_us":290,"threads_started":1}
W20260806 09:22:42.860376 29987 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:42.841568 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:53413 (local address 127.16.86.190:41485)
0806 09:22:42.842337 (+ 769us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:42.842344 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:42.842367 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:22:42.842437 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:42.842444 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:42.842559 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:42.842723 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:42.844079 (+ 1356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:42.845435 (+ 1356us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:42.847588 (+ 2153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:42.848430 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:42.848612 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:42.860028 (+ 11416us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:53413: BlockingRecv error: recv got EOF from 127.16.86.130:53413 (error 108)
Metrics: {"server-negotiator.queue_time_us":581,"thread_start_us":465,"threads_started":1}
Aug 06 09:22:43 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:43 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:43.166743 29989 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:43.155094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:39031)
0806 09:22:43.155855 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:43.155886 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:43.156044 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:43.156440 (+ 396us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:43.156444 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:43.156881 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:43.157073 (+ 192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:43.157082 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:43.158239 (+ 1157us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:43.158243 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:43.159493 (+ 1250us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:43.159508 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:43.160410 (+ 902us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:43.160418 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:43.160599 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:43.161528 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:43.161556 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:43.166431 (+ 4875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":474,"threads_started":1}
W20260806 09:22:43.166966 29987 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:43.155090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:39031 (local address 127.16.86.190:41485)
0806 09:22:43.155345 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:43.155351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:43.155997 (+ 646us) server_negotiation.cc:407] Connection header received
0806 09:22:43.156177 (+ 180us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:43.156181 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:43.156255 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:43.156366 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:43.157240 (+ 874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:43.158091 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:43.159659 (+ 1568us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:43.160164 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:43.160322 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:43.166685 (+ 6363us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:39031: BlockingRecv error: recv got EOF from 127.16.86.129:39031 (error 108)
Metrics: {"server-negotiator.queue_time_us":103}
Aug 06 09:22:43 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:43 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:43.307169 29990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:43.295655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:51099)
0806 09:22:43.296424 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:43.296445 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:43.296644 (+ 199us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:43.297035 (+ 391us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:43.297040 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:43.297571 (+ 531us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:43.297746 (+ 175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:43.297755 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:43.298952 (+ 1197us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:43.298963 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:43.300116 (+ 1153us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:43.300129 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:43.300925 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:43.300931 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:43.301122 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:43.301867 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:43.301887 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:43.306831 (+ 4944us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":646,"thread_start_us":437,"threads_started":1}
W20260806 09:22:43.307833 29987 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:43.295774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:51099 (local address 127.16.86.190:41485)
0806 09:22:43.296132 (+ 358us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:43.296142 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:43.296552 (+ 410us) server_negotiation.cc:407] Connection header received
0806 09:22:43.296797 (+ 245us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:43.296801 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:43.296890 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:43.297003 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:43.297891 (+ 888us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:43.298788 (+ 897us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:43.300265 (+ 1477us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:43.300735 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:43.300877 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:43.307203 (+ 6326us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:51099: BlockingRecv error: recv got EOF from 127.16.86.131:51099 (error 108)
Metrics: {"server-negotiator.queue_time_us":156}
Aug 06 09:22:43 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:43 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:43.873485 29991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:43.861488 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:46475)
0806 09:22:43.862443 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:43.862472 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:43.862652 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:43.863119 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:43.863124 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:43.863629 (+ 505us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:43.863806 (+ 177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:43.863814 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:43.865120 (+ 1306us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:43.865126 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:43.866438 (+ 1312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:43.866452 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:43.867364 (+ 912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:43.867371 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:43.867572 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:43.868337 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:43.868356 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:43.873060 (+ 4704us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":794,"thread_start_us":636,"threads_started":1}
W20260806 09:22:43.873770 29992 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:43.861540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:46475 (local address 127.16.86.190:41485)
0806 09:22:43.862406 (+ 866us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:43.862413 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:43.862608 (+ 195us) server_negotiation.cc:407] Connection header received
0806 09:22:43.862833 (+ 225us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:43.862837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:43.862914 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:43.863074 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:43.863976 (+ 902us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:43.864944 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:43.866595 (+ 1651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:43.867181 (+ 586us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:43.867334 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:43.873446 (+ 6112us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:46475: BlockingRecv error: recv got EOF from 127.16.86.130:46475 (error 108)
Metrics: {"server-negotiator.queue_time_us":715,"thread_start_us":525,"threads_started":1}
Aug 06 09:22:44 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:44 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:44.178838 29993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:44.168204 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:56957)
0806 09:22:44.168886 (+ 682us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:44.168912 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:44.169108 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:44.169507 (+ 399us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:44.169512 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:44.170058 (+ 546us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:44.170264 (+ 206us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:44.170274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:44.171497 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:44.171505 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:44.172488 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:44.172500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:44.173254 (+ 754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:44.173260 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:44.173468 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:44.174201 (+ 733us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:44.174220 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:44.178518 (+ 4298us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":381,"threads_started":1}
W20260806 09:22:44.179148 29992 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:44.168500 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:56957 (local address 127.16.86.190:41485)
0806 09:22:44.168784 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:44.168790 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:44.169028 (+ 238us) server_negotiation.cc:407] Connection header received
0806 09:22:44.169254 (+ 226us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:44.169260 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:44.169348 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:44.169466 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:44.170427 (+ 961us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:44.171360 (+ 933us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:44.172646 (+ 1286us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:44.173092 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:44.173235 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:44.178826 (+ 5591us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:56957: BlockingRecv error: recv got EOF from 127.16.86.129:56957 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Aug 06 09:22:44 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:44 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:44.319705 29994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:44.308370 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:58721)
0806 09:22:44.309006 (+ 636us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:44.309027 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:44.309217 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:44.309792 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:44.309796 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:44.310251 (+ 455us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:44.310461 (+ 210us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:44.310470 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:44.311827 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:44.311831 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:44.312911 (+ 1080us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:44.312924 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:44.313780 (+ 856us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:44.313785 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:44.313919 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:44.314651 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:44.314674 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:44.319381 (+ 4707us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":312,"threads_started":1}
W20260806 09:22:44.319805 29992 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:44.308510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:58721 (local address 127.16.86.190:41485)
0806 09:22:44.308809 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:44.308817 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:44.309167 (+ 350us) server_negotiation.cc:407] Connection header received
0806 09:22:44.309507 (+ 340us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:44.309512 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:44.309613 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:44.309708 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:44.310672 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:44.311646 (+ 974us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:44.313075 (+ 1429us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:44.313572 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:44.313688 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:44.319578 (+ 5890us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:58721: BlockingRecv error: recv got EOF from 127.16.86.131:58721 (error 108)
Metrics: {"server-negotiator.queue_time_us":120}
Aug 06 09:22:44 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:44 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:44.890733 29995 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:44.874906 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:60473)
0806 09:22:44.875947 (+ 1041us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:44.875967 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:44.876173 (+ 206us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:44.876979 (+ 806us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:44.876987 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:44.877732 (+ 745us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:44.878045 (+ 313us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:44.878062 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:44.880045 (+ 1983us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:44.880052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:44.882494 (+ 2442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:44.882521 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:44.883906 (+ 1385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:44.883914 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:44.884150 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:44.885097 (+ 947us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:44.885128 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:44.890442 (+ 5314us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":874,"thread_start_us":738,"threads_started":1}
W20260806 09:22:44.891204 29996 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:44.875077 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:60473 (local address 127.16.86.190:41485)
0806 09:22:44.875787 (+ 710us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:44.875796 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:44.876074 (+ 278us) server_negotiation.cc:407] Connection header received
0806 09:22:44.876354 (+ 280us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:44.876362 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:44.876503 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:44.876677 (+ 174us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:44.878293 (+ 1616us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:44.879866 (+ 1573us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:44.882705 (+ 2839us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:44.883704 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:44.883920 (+ 216us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:44.890734 (+ 6814us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:60473: BlockingRecv error: recv got EOF from 127.16.86.130:60473 (error 108)
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":331,"threads_started":1}
Aug 06 09:22:45 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:45 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:45.192082 29998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:45.180217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:45435)
0806 09:22:45.180920 (+ 703us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:45.180946 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:45.181121 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:45.181496 (+ 375us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:45.181501 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:45.182091 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:45.182319 (+ 228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:45.182331 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:45.183920 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:45.183924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:45.184997 (+ 1073us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:45.185010 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:45.185979 (+ 969us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:45.185985 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:45.186197 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:45.187228 (+ 1031us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:45.187255 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:45.191737 (+ 4482us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":412,"threads_started":1}
W20260806 09:22:45.192308 29996 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:45.180393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:45435 (local address 127.16.86.190:41485)
0806 09:22:45.180716 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:45.180723 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:45.181067 (+ 344us) server_negotiation.cc:407] Connection header received
0806 09:22:45.181250 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:45.181254 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:45.181335 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:45.181433 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:45.182512 (+ 1079us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:45.183760 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:45.185159 (+ 1399us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:45.185781 (+ 622us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:45.185919 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:45.192001 (+ 6082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:45435: BlockingRecv error: recv got EOF from 127.16.86.129:45435 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Aug 06 09:22:45 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:45 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:45.333447 30000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:45.321199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:41919)
0806 09:22:45.321917 (+ 718us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:45.321938 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:45.322093 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:45.322543 (+ 450us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:45.322547 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:45.323163 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:45.323435 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:45.323446 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:45.324580 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:45.324585 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:45.325757 (+ 1172us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:45.325771 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:45.326859 (+ 1088us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:45.326895 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:45.327133 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:45.328113 (+ 980us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:45.328156 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:45.333133 (+ 4977us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":555,"thread_start_us":408,"threads_started":1}
W20260806 09:22:45.334159 29996 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:45.321409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:41919 (local address 127.16.86.190:41485)
0806 09:22:45.321638 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:45.321643 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:45.322067 (+ 424us) server_negotiation.cc:407] Connection header received
0806 09:22:45.322281 (+ 214us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:45.322286 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:45.322365 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:45.322492 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:45.323631 (+ 1139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:45.324436 (+ 805us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:45.325937 (+ 1501us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:45.326624 (+ 687us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:45.326799 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:45.333483 (+ 6684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:41919: BlockingRecv error: recv got EOF from 127.16.86.131:41919 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Aug 06 09:22:45 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:45 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:45.904682 30001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:45.891947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:43573)
0806 09:22:45.892829 (+ 882us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:45.893074 (+ 245us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:45.893250 (+ 176us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:45.893695 (+ 445us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:45.893700 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:45.894330 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:45.894618 (+ 288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:45.894637 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:45.896354 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:45.896360 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:45.897482 (+ 1122us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:45.897499 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:45.898370 (+ 871us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:45.898377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:45.898607 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:45.899437 (+ 830us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:45.899458 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:45.904323 (+ 4865us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":699,"thread_start_us":560,"threads_started":1}
W20260806 09:22:45.905023 30002 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:45.892050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:43573 (local address 127.16.86.190:41485)
0806 09:22:45.892613 (+ 563us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:45.892618 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:45.893205 (+ 587us) server_negotiation.cc:407] Connection header received
0806 09:22:45.893393 (+ 188us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:45.893397 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:45.893489 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:45.893609 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:45.894792 (+ 1183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:45.896095 (+ 1303us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:45.897678 (+ 1583us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:45.898194 (+ 516us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:45.898347 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:45.904609 (+ 6262us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:43573: BlockingRecv error: recv got EOF from 127.16.86.130:43573 (error 108)
Metrics: {"server-negotiator.queue_time_us":451,"thread_start_us":323,"threads_started":1}
Aug 06 09:22:46 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:46 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:46.204555 30003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:46.193408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:52405)
0806 09:22:46.194070 (+ 662us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:46.194093 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:46.194286 (+ 193us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:46.194652 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:46.194657 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:46.195197 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:46.195354 (+ 157us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:46.195364 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:46.196403 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:46.196409 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:46.197340 (+ 931us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:46.197352 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:46.198101 (+ 749us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:46.198116 (+ 15us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:46.198294 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:46.199033 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:46.199052 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:46.204142 (+ 5090us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":370,"threads_started":1}
W20260806 09:22:46.204666 30002 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:46.193517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:52405 (local address 127.16.86.190:41485)
0806 09:22:46.193816 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:46.193824 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:46.194210 (+ 386us) server_negotiation.cc:407] Connection header received
0806 09:22:46.194427 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:46.194433 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:46.194507 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:46.194618 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:46.195503 (+ 885us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:46.196262 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:46.197488 (+ 1226us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:46.197911 (+ 423us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:46.198063 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:46.204410 (+ 6347us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:52405: BlockingRecv error: recv got EOF from 127.16.86.129:52405 (error 108)
Metrics: {"server-negotiator.queue_time_us":126}
Aug 06 09:22:46 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:46 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:46.346558 30007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:46.334655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:59879)
0806 09:22:46.335416 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:46.335436 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:46.335602 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:46.336102 (+ 500us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:46.336105 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:46.336646 (+ 541us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:46.336952 (+ 306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:46.336965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:46.338188 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:46.338193 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:46.339323 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:46.339338 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:46.340185 (+ 847us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:46.340190 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:46.340364 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:46.341141 (+ 777us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:46.341161 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:46.346208 (+ 5047us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":611,"thread_start_us":479,"threads_started":1}
W20260806 09:22:46.347038 30002 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:46.334811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:59879 (local address 127.16.86.190:41485)
0806 09:22:46.335076 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:46.335082 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:46.335635 (+ 553us) server_negotiation.cc:407] Connection header received
0806 09:22:46.335802 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:46.335806 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:46.335880 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:46.336002 (+ 122us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:46.337150 (+ 1148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:46.337992 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:46.339506 (+ 1514us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:46.339975 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:46.340094 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:46.346529 (+ 6435us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:59879: BlockingRecv error: recv got EOF from 127.16.86.131:59879 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
Aug 06 09:22:46 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:46 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.130@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:46.917644 30008 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:46.905968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.130:48475)
0806 09:22:46.906638 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:46.906668 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:46.906815 (+ 147us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:46.907217 (+ 402us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:46.907221 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:46.907702 (+ 481us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:46.907885 (+ 183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:46.907893 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:46.908965 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:46.908971 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:46.910330 (+ 1359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:46.910345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:46.911253 (+ 908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:46.911261 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:46.911487 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:46.912585 (+ 1098us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:46.912628 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:46.917256 (+ 4628us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":495,"thread_start_us":325,"threads_started":1}
W20260806 09:22:46.917804 30009 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:46.906209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:48475 (local address 127.16.86.190:41485)
0806 09:22:46.906832 (+ 623us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:46.906837 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:46.906885 (+ 48us) server_negotiation.cc:407] Connection header received
0806 09:22:46.906936 (+ 51us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:46.906940 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:46.907064 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:46.907171 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:46.908021 (+ 850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:46.908780 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:46.910483 (+ 1703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:46.911046 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:46.911208 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:46.917502 (+ 6294us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.130:48475: BlockingRecv error: recv got EOF from 127.16.86.130:48475 (error 108)
Metrics: {"server-negotiator.queue_time_us":506,"thread_start_us":321,"threads_started":1}
Aug 06 09:22:47 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:47 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.129@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:47.216599 30012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:47.206074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.129:54093)
0806 09:22:47.206711 (+ 637us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:47.206732 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:47.206896 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:47.207301 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:47.207306 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:47.207810 (+ 504us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:47.207992 (+ 182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:47.208001 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.209128 (+ 1127us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.209133 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:47.210212 (+ 1079us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:47.210225 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.210912 (+ 687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.210918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:47.211119 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:47.211911 (+ 792us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:47.211930 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:47.216217 (+ 4287us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":331,"threads_started":1}
W20260806 09:22:47.216696 30009 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:47.206185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:54093 (local address 127.16.86.190:41485)
0806 09:22:47.206439 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:47.206447 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:47.206840 (+ 393us) server_negotiation.cc:407] Connection header received
0806 09:22:47.207075 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:47.207079 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:47.207160 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:47.207270 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:47.208147 (+ 877us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.208978 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.210352 (+ 1374us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.210762 (+ 410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.210895 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:47.216431 (+ 5536us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.129:54093: BlockingRecv error: recv got EOF from 127.16.86.129:54093 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
Aug 06 09:22:47 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
Aug 06 09:22:47 dist-test-slave-h3ft krb5kdc[28471](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.16.86.131@KRBTEST.COM for oryx/127.16.86.190@KRBTEST.COM, Server not found in Kerberos database
I20260806 09:22:47.360402 30013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:47.347762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41485 (local address 127.16.86.131:43865)
0806 09:22:47.348475 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:47.348495 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:47.348698 (+ 203us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:47.349106 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:47.349112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:47.349708 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:47.349898 (+ 190us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:47.349918 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.351275 (+ 1357us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.351282 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:47.352371 (+ 1089us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:47.352392 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.353210 (+ 818us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.353217 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:47.353430 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:47.354313 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:47.354338 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:47.360052 (+ 5714us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.16.86.190:41485: Server oryx/127.16.86.190@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":371,"threads_started":1}
W20260806 09:22:47.360750 30009 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:47.347888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:43865 (local address 127.16.86.190:41485)
0806 09:22:47.348336 (+ 448us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:47.348344 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:47.348599 (+ 255us) server_negotiation.cc:407] Connection header received
0806 09:22:47.348846 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:47.348856 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:47.348943 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:47.349061 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:47.350079 (+ 1018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.351121 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.352541 (+ 1420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.353043 (+ 502us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.353193 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:47.360402 (+ 7209us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.16.86.131:43865: BlockingRecv error: recv got EOF from 127.16.86.131:43865 (error 108)
Metrics: {"server-negotiator.queue_time_us":243}
I20260806 09:22:47.828536 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 29331
I20260806 09:22:47.844632 30015 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:47.829987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35416 (local address 127.16.86.129:44991)
0806 09:22:47.830743 (+ 756us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:47.830751 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:47.830781 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:22:47.830864 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:47.830872 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:47.831181 (+ 309us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:47.831379 (+ 198us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:47.833268 (+ 1889us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.834344 (+ 1076us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.835860 (+ 1516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.836420 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.836541 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:47.839768 (+ 3227us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:47.839778 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:47.839792 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:47.839838 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:47.843657 (+ 3819us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.86.129@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 09:22:47.844218 (+ 561us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:35416: Request ticket server kudu/127.16.86.129@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":517,"thread_start_us":398,"threads_started":1}
W20260806 09:22:47.844794 30015 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:35416: Request ticket server kudu/127.16.86.129@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 09:22:47.844767 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 29331: Not authorized: Client connection negotiation failed: client connection to 127.16.86.129:44991: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.86.129@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 09:22:47.845065 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 29331
I20260806 09:22:47.880035 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 29541
I20260806 09:22:47.903913 30016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:47.881811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.16.86.130:35661)
0806 09:22:47.882574 (+ 763us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:47.882584 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:47.882618 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:22:47.882711 (+ 93us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:47.882725 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:47.883169 (+ 444us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:47.883435 (+ 266us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:47.885016 (+ 1581us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.886173 (+ 1157us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.887590 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.888293 (+ 703us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.888454 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:47.898907 (+ 10453us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:47.898922 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:47.898944 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:47.899103 (+ 159us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:47.902807 (+ 3704us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.86.130@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 09:22:47.903437 (+ 630us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:40910: Request ticket server kudu/127.16.86.130@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":381,"threads_started":1}
W20260806 09:22:47.904069 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 29541: Not authorized: Client connection negotiation failed: client connection to 127.16.86.130:35661: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.86.130@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 09:22:47.904107 30016 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:40910: Request ticket server kudu/127.16.86.130@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 09:22:47.904281 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 29541
I20260806 09:22:47.934904 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 29735
I20260806 09:22:47.951643 30019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:47.936590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42640 (local address 127.16.86.131:42121)
0806 09:22:47.937414 (+ 824us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:47.937426 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:47.937462 (+ 36us) server_negotiation.cc:407] Connection header received
0806 09:22:47.937554 (+ 92us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:47.937563 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:47.937889 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:47.938187 (+ 298us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:47.940141 (+ 1954us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.941533 (+ 1392us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.943587 (+ 2054us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:47.944265 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:47.944447 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:47.947679 (+ 3232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:47.947690 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:47.947706 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:47.947757 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:47.950794 (+ 3037us) server_negotiation.cc:394] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.86.131@KRBTEST.COM not found in keytab (ticket kvno 2)
0806 09:22:47.951271 (+ 477us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42640: Request ticket server kudu/127.16.86.131@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":397,"threads_started":1}
W20260806 09:22:47.951794 30019 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42640: Request ticket server kudu/127.16.86.131@KRBTEST.COM not found in keytab (ticket kvno 2)
W20260806 09:22:47.951805 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 29735: Not authorized: Client connection negotiation failed: client connection to 127.16.86.131:42121: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.16.86.131@KRBTEST.COM not found in keytab (ticket kvno 2)
I20260806 09:22:47.951992 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 29735
I20260806 09:22:47.983224 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 29225
I20260806 09:22:47.984915 29318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:48.129122 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 29225
2026-08-06T09:22:48Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (22870 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2026-08-06T09:22:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:22:48Z Disabled control of system clock
I20260806 09:22:48.243657 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:39631
--webserver_interface=127.16.86.190
--webserver_port=0
--builtin_ntp_servers=127.16.86.148:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:39631
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20260806 09:22:48.627849 30029 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:48.628177 30029 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:48.628265 30029 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:48.635679 30029 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:22:48.635800 30029 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:48.635840 30029 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:48.635890 30029 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:22:48.635959 30029 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:22:48.646168 30029 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:39631
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:39631
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:48.647660 30029 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:48.649718 30029 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:48.663862 30041 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:48.664004 30038 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:48.664309 30037 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:48.665508 30029 server_base.cc:1034] running on GCE node
I20260806 09:22:48.666890 30029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:48.668653 30029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:48.669977 30029 hybrid_clock.cc:648] HybridClock initialized: now 1786008168669906 us; error 93 us; skew 500 ppm
I20260806 09:22:48.670476 30029 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:22:48.673434 30029 webserver.cc:460] Webserver started at http://127.16.86.190:34039/ using document root <none> and password file <none>
I20260806 09:22:48.674386 30029 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:48.674542 30029 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:48.674911 30029 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:48.678653 30029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "e934a034ed1d4845beaf213d318fa873"
format_stamp: "Formatted at 2026-08-06 09:22:48 on dist-test-slave-h3ft"
server_key: "3c479d02e7ad4a4107fa2e5e8f100508"
server_key_iv: "0429ec771c85dbd0dc523e21192fb52e"
server_key_version: "encryptionkey@0"
I20260806 09:22:48.679567 30029 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "e934a034ed1d4845beaf213d318fa873"
format_stamp: "Formatted at 2026-08-06 09:22:48 on dist-test-slave-h3ft"
server_key: "3c479d02e7ad4a4107fa2e5e8f100508"
server_key_iv: "0429ec771c85dbd0dc523e21192fb52e"
server_key_version: "encryptionkey@0"
I20260806 09:22:48.686187 30029 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260806 09:22:48.690868 30048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:48.692785 30029 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:22:48.693046 30029 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "e934a034ed1d4845beaf213d318fa873"
format_stamp: "Formatted at 2026-08-06 09:22:48 on dist-test-slave-h3ft"
server_key: "3c479d02e7ad4a4107fa2e5e8f100508"
server_key_iv: "0429ec771c85dbd0dc523e21192fb52e"
server_key_version: "encryptionkey@0"
I20260806 09:22:48.693312 30029 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:48.726222 30029 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:48.727279 30029 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:48.727613 30029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:48.750898 30029 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:39631
I20260806 09:22:48.751086 30114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:39631 every 8 connection(s)
I20260806 09:22:48.752529 30029 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:22:48.756574 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 30029
I20260806 09:22:48.756840 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:22:48.757824 16730 external_mini_cluster.cc:1428] Setting key 166db728cd87606b2dd00474a53a2f22
I20260806 09:22:48.758490 30115 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:48.776012 30119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:48.760753 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33616 (local address 127.16.86.190:39631)
0806 09:22:48.761985 (+ 1232us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:48.761998 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:48.762050 (+ 52us) server_negotiation.cc:407] Connection header received
0806 09:22:48.764505 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:48.764530 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:48.765127 (+ 597us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:48.765512 (+ 385us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:48.766462 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:48.768052 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:48.770107 (+ 2055us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:48.770846 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:48.771088 (+ 242us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:48.772761 (+ 1673us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:48.772770 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:48.772789 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:48.773730 (+ 941us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:48.773839 (+ 109us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:48.773863 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:48.773970 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:48.774259 (+ 289us) server_negotiation.cc:299] Negotiation successful
0806 09:22:48.774788 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":706,"thread_start_us":452,"threads_started":1}
I20260806 09:22:48.776515 30115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873: Bootstrap starting.
I20260806 09:22:48.780800 30115 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:48.782116 30115 log.cc:824] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:48.785892 30115 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873: No bootstrap required, opened a new log
I20260806 09:22:48.794152 30115 raft_consensus.cc:348] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e934a034ed1d4845beaf213d318fa873" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 39631 } }
I20260806 09:22:48.794632 30115 raft_consensus.cc:374] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:48.794742 30115 raft_consensus.cc:729] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e934a034ed1d4845beaf213d318fa873, State: Initialized, Role: FOLLOWER
I20260806 09:22:48.795526 30115 consensus_queue.cc:260] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e934a034ed1d4845beaf213d318fa873" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 39631 } }
I20260806 09:22:48.795790 30115 raft_consensus.cc:388] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:22:48.795969 30115 raft_consensus.cc:482] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:22:48.796196 30115 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:48.799422 30115 raft_consensus.cc:504] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e934a034ed1d4845beaf213d318fa873" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 39631 } }
I20260806 09:22:48.800103 30115 leader_election.cc:302] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e934a034ed1d4845beaf213d318fa873; no voters:
I20260806 09:22:48.801160 30115 leader_election.cc:288] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:22:48.801389 30121 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:48.802367 30121 raft_consensus.cc:686] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [term 1 LEADER]: Becoming Leader. State: Replica: e934a034ed1d4845beaf213d318fa873, State: Running, Role: LEADER
I20260806 09:22:48.803143 30121 consensus_queue.cc:237] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e934a034ed1d4845beaf213d318fa873" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 39631 } }
I20260806 09:22:48.803989 30115 sys_catalog.cc:564] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:22:48.806447 30123 sys_catalog.cc:455] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e934a034ed1d4845beaf213d318fa873. Latest consensus state: current_term: 1 leader_uuid: "e934a034ed1d4845beaf213d318fa873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e934a034ed1d4845beaf213d318fa873" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 39631 } } }
I20260806 09:22:48.807075 30123 sys_catalog.cc:458] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:48.808130 30122 sys_catalog.cc:455] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e934a034ed1d4845beaf213d318fa873" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e934a034ed1d4845beaf213d318fa873" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 39631 } } }
I20260806 09:22:48.808560 30122 sys_catalog.cc:458] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873 [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:48.811399 30128 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:22:48.817307 30128 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:22:48.828559 30128 catalog_manager.cc:1347] Generated new cluster ID: f2e1975b9a3e48d4ab1d625e926d2ba7
I20260806 09:22:48.828729 30128 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:22:48.864884 30128 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:22:48.866603 30128 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:22:48.875844 30128 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e934a034ed1d4845beaf213d318fa873: Generated new TSK 0
I20260806 09:22:48.876860 30128 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:22:48.896611 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:39631
--builtin_ntp_servers=127.16.86.148:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20260806 09:22:49.278108 30146 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:49.278440 30146 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:49.278597 30146 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:49.286098 30146 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:49.286255 30146 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:49.286432 30146 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:22:49.295750 30146 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:39631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:49.297145 30146 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:49.299098 30146 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:49.314196 30160 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:49.314237 30156 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:49.315069 30157 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:49.316169 30146 server_base.cc:1034] running on GCE node
I20260806 09:22:49.316841 30146 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:49.317982 30146 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:49.319226 30146 hybrid_clock.cc:648] HybridClock initialized: now 1786008169319140 us; error 87 us; skew 500 ppm
I20260806 09:22:49.319566 30146 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:22:49.322165 30146 webserver.cc:460] Webserver started at http://127.16.86.129:34689/ using document root <none> and password file <none>
I20260806 09:22:49.322947 30146 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:49.323133 30146 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:49.323495 30146 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:49.326401 30146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "10102db20c4d4bee962658d647c0a5c1"
format_stamp: "Formatted at 2026-08-06 09:22:49 on dist-test-slave-h3ft"
server_key: "7c559bbdc4532bd22605be2600950d05"
server_key_iv: "fad644af288f7d3667322d3d6cf1b048"
server_key_version: "encryptionkey@0"
I20260806 09:22:49.327306 30146 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "10102db20c4d4bee962658d647c0a5c1"
format_stamp: "Formatted at 2026-08-06 09:22:49 on dist-test-slave-h3ft"
server_key: "7c559bbdc4532bd22605be2600950d05"
server_key_iv: "fad644af288f7d3667322d3d6cf1b048"
server_key_version: "encryptionkey@0"
I20260806 09:22:49.333053 30146 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260806 09:22:49.337066 30172 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:49.338712 30146 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260806 09:22:49.338922 30146 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "10102db20c4d4bee962658d647c0a5c1"
format_stamp: "Formatted at 2026-08-06 09:22:49 on dist-test-slave-h3ft"
server_key: "7c559bbdc4532bd22605be2600950d05"
server_key_iv: "fad644af288f7d3667322d3d6cf1b048"
server_key_version: "encryptionkey@0"
I20260806 09:22:49.339207 30146 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:49.354198 30146 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:49.355140 30146 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:49.355473 30146 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:49.356633 30146 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:49.358256 30146 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:49.358389 30146 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:49.358541 30146 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:49.358644 30146 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:49.402499 30146 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:44015
I20260806 09:22:49.402539 30332 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:44015 every 8 connection(s)
I20260806 09:22:49.404016 30146 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:22:49.407770 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 30146
I20260806 09:22:49.407979 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:22:49.408757 16730 external_mini_cluster.cc:1428] Setting key 567fb197ee7901f80c2f940c2abf272f
I20260806 09:22:49.412613 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:39631
--builtin_ntp_servers=127.16.86.148:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 09:22:49.421533 30337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:49.407810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:38655 (local address 127.16.86.190:39631)
0806 09:22:49.408451 (+ 641us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:49.408459 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:49.410253 (+ 1794us) server_negotiation.cc:407] Connection header received
0806 09:22:49.412823 (+ 2570us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:49.412832 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:49.412954 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:49.413082 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:49.414667 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:49.415591 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:49.418134 (+ 2543us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:49.418580 (+ 446us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:49.418699 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:49.420565 (+ 1866us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:49.420571 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:49.420574 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:49.420600 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:49.420651 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:49.420664 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:49.420757 (+ 93us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:49.421088 (+ 331us) server_negotiation.cc:299] Negotiation successful
0806 09:22:49.421196 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":318,"threads_started":1}
I20260806 09:22:49.422950 30340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:49.408666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:39631 (local address 127.16.86.129:38655)
0806 09:22:49.410033 (+ 1367us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:49.410096 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:49.412365 (+ 2269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:49.413386 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:49.413401 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:49.413533 (+ 132us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:49.414454 (+ 921us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:49.414481 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:49.415780 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:49.415789 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:49.417948 (+ 2159us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:49.417967 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:49.418841 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:49.418851 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:49.419231 (+ 380us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:49.420138 (+ 907us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:22:49.420144 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:49.420202 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:22:49.420311 (+ 109us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:49.420788 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:49.420803 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:49.420814 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 09:22:49.421026 (+ 212us) client_negotiation.cc:240] Negotiation successful
0806 09:22:49.421475 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":653,"thread_start_us":380,"threads_started":1}
I20260806 09:22:49.425170 30334 heartbeater.cc:347] Connected to a master server at 127.16.86.190:39631
I20260806 09:22:49.425781 30334 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:49.427386 30334 heartbeater.cc:511] Master 127.16.86.190:39631 requested a full tablet report, sending...
I20260806 09:22:49.430220 30071 ts_manager.cc:194] Registered new tserver with Master: 10102db20c4d4bee962658d647c0a5c1 (127.16.86.129:44015)
I20260806 09:22:49.431880 30071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.86.129:38655
W20260806 09:22:49.840083 30341 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:49.840504 30341 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:49.840678 30341 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:49.849913 30341 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:49.850069 30341 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:49.850266 30341 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:22:49.861002 30341 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:39631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:49.863059 30341 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:49.865393 30341 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:49.881916 30350 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:49.881850 30353 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:49.882663 30348 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:49.883603 30341 server_base.cc:1034] running on GCE node
I20260806 09:22:49.884459 30341 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:49.885658 30341 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:49.886930 30341 hybrid_clock.cc:648] HybridClock initialized: now 1786008169886832 us; error 93 us; skew 500 ppm
I20260806 09:22:49.887360 30341 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:22:49.890081 30341 webserver.cc:460] Webserver started at http://127.16.86.130:40183/ using document root <none> and password file <none>
I20260806 09:22:49.890883 30341 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:49.891049 30341 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:49.891426 30341 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:49.894618 30341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "b703f46083754fd58efd028e6ee47c19"
format_stamp: "Formatted at 2026-08-06 09:22:49 on dist-test-slave-h3ft"
server_key: "2fba08863138fbd398ca55ed23bbed36"
server_key_iv: "8a4403d82f3d1d0e2916b21d89bc43f0"
server_key_version: "encryptionkey@0"
I20260806 09:22:49.895507 30341 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "b703f46083754fd58efd028e6ee47c19"
format_stamp: "Formatted at 2026-08-06 09:22:49 on dist-test-slave-h3ft"
server_key: "2fba08863138fbd398ca55ed23bbed36"
server_key_iv: "8a4403d82f3d1d0e2916b21d89bc43f0"
server_key_version: "encryptionkey@0"
I20260806 09:22:49.901861 30341 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:22:49.906700 30361 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:49.908681 30341 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 09:22:49.908946 30341 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "b703f46083754fd58efd028e6ee47c19"
format_stamp: "Formatted at 2026-08-06 09:22:49 on dist-test-slave-h3ft"
server_key: "2fba08863138fbd398ca55ed23bbed36"
server_key_iv: "8a4403d82f3d1d0e2916b21d89bc43f0"
server_key_version: "encryptionkey@0"
I20260806 09:22:49.909265 30341 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:49.931106 30341 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:49.932153 30341 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:49.932528 30341 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:49.933794 30341 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:49.935722 30341 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:49.935865 30341 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:49.936033 30341 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:49.936146 30341 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:49.980693 30341 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:39685
I20260806 09:22:49.980849 30518 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:39685 every 8 connection(s)
I20260806 09:22:49.982656 30341 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:22:49.984558 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 30341
I20260806 09:22:49.984766 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:22:49.985404 16730 external_mini_cluster.cc:1428] Setting key 059022ac1b12d1f9b2e07fc70991c71c
I20260806 09:22:49.989344 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:39631
--builtin_ntp_servers=127.16.86.148:46135
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20260806 09:22:50.003906 30523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:49.986963 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:40595 (local address 127.16.86.190:39631)
0806 09:22:49.987636 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:49.987642 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:49.989810 (+ 2168us) server_negotiation.cc:407] Connection header received
0806 09:22:49.993336 (+ 3526us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:49.993341 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:49.993441 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:49.993591 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:49.995335 (+ 1744us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:49.996394 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:49.999257 (+ 2863us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:49.999853 (+ 596us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:49.999999 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:50.002351 (+ 2352us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:50.002360 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:50.002364 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:50.002396 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:50.002477 (+ 81us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:50.002496 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:50.002598 (+ 102us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:50.003149 (+ 551us) server_negotiation.cc:299] Negotiation successful
0806 09:22:50.003345 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":355,"threads_started":1}
I20260806 09:22:50.004707 30524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:49.987948 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:39631 (local address 127.16.86.130:40595)
0806 09:22:49.989560 (+ 1612us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:49.989653 (+ 93us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:49.992903 (+ 3250us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:49.993924 (+ 1021us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:49.993940 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:49.994086 (+ 146us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:49.995062 (+ 976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:49.995092 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:49.996618 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:49.996627 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:49.998993 (+ 2366us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:49.999052 (+ 59us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:50.000207 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:50.000218 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:50.000600 (+ 382us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:50.001610 (+ 1010us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:22:50.001616 (+ 6us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:50.001679 (+ 63us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:22:50.001802 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:50.002645 (+ 843us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:50.002659 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:50.002671 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 09:22:50.002905 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 09:22:50.003366 (+ 461us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":726,"thread_start_us":400,"threads_started":1}
I20260806 09:22:50.006631 30519 heartbeater.cc:347] Connected to a master server at 127.16.86.190:39631
I20260806 09:22:50.006995 30519 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:50.007965 30519 heartbeater.cc:511] Master 127.16.86.190:39631 requested a full tablet report, sending...
I20260806 09:22:50.009827 30071 ts_manager.cc:194] Registered new tserver with Master: b703f46083754fd58efd028e6ee47c19 (127.16.86.130:39685)
I20260806 09:22:50.010903 30071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.86.130:40595
W20260806 09:22:50.431905 30525 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:50.432309 30525 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:50.432411 30525 flags.cc:425] Enabled unsafe flag: --never_fsync=true
I20260806 09:22:50.436875 30334 heartbeater.cc:503] Master 127.16.86.190:39631 was elected leader, sending a full tablet report...
W20260806 09:22:50.441308 30525 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:50.441445 30525 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:50.441567 30525 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:22:50.452256 30525 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46135
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:39631
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:50.454066 30525 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:50.456420 30525 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:50.472783 30533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:50.472684 30534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:50.472829 30536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:50.474804 30525 server_base.cc:1034] running on GCE node
I20260806 09:22:50.475847 30525 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:50.477324 30525 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:50.478590 30525 hybrid_clock.cc:648] HybridClock initialized: now 1786008170478501 us; error 75 us; skew 500 ppm
I20260806 09:22:50.479125 30525 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:22:50.482363 30525 webserver.cc:460] Webserver started at http://127.16.86.131:37849/ using document root <none> and password file <none>
I20260806 09:22:50.483394 30525 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:50.483538 30525 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:50.483969 30525 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:50.487746 30525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "68ad57319e6444bdaf4e68589bd40601"
format_stamp: "Formatted at 2026-08-06 09:22:50 on dist-test-slave-h3ft"
server_key: "42669e5fc2a42cdc84cb9d23abebe84b"
server_key_iv: "46e746d40f204934c67320451d1c3a5f"
server_key_version: "encryptionkey@0"
I20260806 09:22:50.488663 30525 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "68ad57319e6444bdaf4e68589bd40601"
format_stamp: "Formatted at 2026-08-06 09:22:50 on dist-test-slave-h3ft"
server_key: "42669e5fc2a42cdc84cb9d23abebe84b"
server_key_iv: "46e746d40f204934c67320451d1c3a5f"
server_key_version: "encryptionkey@0"
I20260806 09:22:50.495620 30525 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260806 09:22:50.500651 30544 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:50.502745 30525 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:22:50.503064 30525 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "68ad57319e6444bdaf4e68589bd40601"
format_stamp: "Formatted at 2026-08-06 09:22:50 on dist-test-slave-h3ft"
server_key: "42669e5fc2a42cdc84cb9d23abebe84b"
server_key_iv: "46e746d40f204934c67320451d1c3a5f"
server_key_version: "encryptionkey@0"
I20260806 09:22:50.503386 30525 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:50.522835 30525 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:50.523890 30525 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:50.524250 30525 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:50.525455 30525 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:50.527704 30525 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:50.527892 30525 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:50.528184 30525 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:50.528343 30525 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:50.572136 30525 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:37785
I20260806 09:22:50.572328 30704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:37785 every 8 connection(s)
I20260806 09:22:50.573800 30525 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:22:50.583113 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 30525
I20260806 09:22:50.583330 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:22:50.583904 16730 external_mini_cluster.cc:1428] Setting key 684cb475e88e06f6aee1b70981c1c261
I20260806 09:22:50.591933 30707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:50.577672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:34819 (local address 127.16.86.190:39631)
0806 09:22:50.578345 (+ 673us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:50.578351 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:50.580192 (+ 1841us) server_negotiation.cc:407] Connection header received
0806 09:22:50.582578 (+ 2386us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:50.582584 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:50.582711 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:50.582853 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:50.584435 (+ 1582us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:50.585384 (+ 949us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:50.588231 (+ 2847us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:50.588814 (+ 583us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:50.588968 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:50.590800 (+ 1832us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:50.590806 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:50.590809 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:50.590829 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:50.590880 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:50.590894 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:50.591040 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:50.591335 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 09:22:50.591456 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":510,"thread_start_us":340,"threads_started":1}
I20260806 09:22:50.593407 30708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:50.578470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:39631 (local address 127.16.86.131:34819)
0806 09:22:50.579928 (+ 1458us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:50.579990 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:50.582175 (+ 2185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:50.583188 (+ 1013us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:50.583198 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:50.583393 (+ 195us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:50.584238 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:50.584260 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:50.585671 (+ 1411us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:50.585681 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:50.588016 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:50.588041 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:50.589110 (+ 1069us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:50.589120 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:50.589421 (+ 301us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:50.590306 (+ 885us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:22:50.590309 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:50.590356 (+ 47us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:22:50.590453 (+ 97us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:50.591041 (+ 588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:50.591056 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:50.591068 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 09:22:50.591304 (+ 236us) client_negotiation.cc:240] Negotiation successful
0806 09:22:50.591879 (+ 575us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":783,"thread_start_us":521,"threads_started":1}
I20260806 09:22:50.595217 30705 heartbeater.cc:347] Connected to a master server at 127.16.86.190:39631
I20260806 09:22:50.595717 30705 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:50.596880 30705 heartbeater.cc:511] Master 127.16.86.190:39631 requested a full tablet report, sending...
I20260806 09:22:50.598991 30071 ts_manager.cc:194] Registered new tserver with Master: 68ad57319e6444bdaf4e68589bd40601 (127.16.86.131:37785)
I20260806 09:22:50.600099 30071 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.86.131:34819
I20260806 09:22:50.602185 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 09:22:50.610563 30707 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:50.608306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.16.86.190:39631)
0806 09:22:50.608629 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:50.608636 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:50.609380 (+ 744us) server_negotiation.cc:407] Connection header received
0806 09:22:50.609493 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:50.609498 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:50.609592 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:50.609744 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:50.610287 (+ 543us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33630: BlockingRecv error: recv got EOF from 127.0.0.1:33630 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:22:50.611908 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 30146
I20260806 09:22:50.622478 30717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:50.613365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58114 (local address 127.16.86.129:44015)
0806 09:22:50.614162 (+ 797us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:50.614170 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:50.614209 (+ 39us) server_negotiation.cc:407] Connection header received
0806 09:22:50.614297 (+ 88us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:50.614308 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:50.614754 (+ 446us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:50.614945 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:50.615531 (+ 586us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:50.616726 (+ 1195us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:50.618210 (+ 1484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:50.618923 (+ 713us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:50.619147 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:50.620537 (+ 1390us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:50.620547 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:50.620572 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:50.621360 (+ 788us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:50.621426 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:50.621449 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:50.621560 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:50.621805 (+ 245us) server_negotiation.cc:299] Negotiation successful
0806 09:22:50.622028 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":389,"threads_started":1}
I20260806 09:22:50.624430 30328 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:50.714649 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 30146
I20260806 09:22:50.746635 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 30341
I20260806 09:22:50.757300 30719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:50.747999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59700 (local address 127.16.86.130:39685)
0806 09:22:50.748722 (+ 723us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:50.748731 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:50.748769 (+ 38us) server_negotiation.cc:407] Connection header received
0806 09:22:50.748860 (+ 91us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:50.748869 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:50.749166 (+ 297us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:50.749357 (+ 191us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:50.749822 (+ 465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:50.750999 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:50.753074 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:50.753840 (+ 766us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:50.753973 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:50.755388 (+ 1415us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:50.755397 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:50.755416 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:50.756222 (+ 806us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:50.756287 (+ 65us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:50.756322 (+ 35us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:50.756428 (+ 106us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:50.756622 (+ 194us) server_negotiation.cc:299] Negotiation successful
0806 09:22:50.756793 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":530,"thread_start_us":348,"threads_started":1}
I20260806 09:22:50.759045 30512 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:50.857962 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 30341
I20260806 09:22:50.890363 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 30525
I20260806 09:22:50.901010 30722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:50.892007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59734 (local address 127.16.86.131:37785)
0806 09:22:50.892939 (+ 932us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:50.892947 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:50.892984 (+ 37us) server_negotiation.cc:407] Connection header received
0806 09:22:50.893051 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:50.893058 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:50.893328 (+ 270us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:50.893521 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:50.894050 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:50.895233 (+ 1183us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:50.896516 (+ 1283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:50.897267 (+ 751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:50.897421 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:50.898710 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:50.898719 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:50.898734 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:50.899887 (+ 1153us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:50.899950 (+ 63us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:50.899975 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:50.900155 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:50.900403 (+ 248us) server_negotiation.cc:299] Negotiation successful
0806 09:22:50.900568 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":709,"thread_start_us":469,"threads_started":1}
I20260806 09:22:50.902500 30700 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:50.997437 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 30525
I20260806 09:22:51.030028 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 30029
I20260806 09:22:51.031795 30110 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:51.140753 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 30029
2026-08-06T09:22:51Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (3003 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2026-08-06T09:22:51Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:22:51Z Disabled control of system clock
I20260806 09:22:51.255935 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:42449
--webserver_interface=127.16.86.190
--webserver_port=0
--builtin_ntp_servers=127.16.86.148:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:42449
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 09:22:51.685182 30728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:51.685609 30728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:51.685719 30728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:51.694291 30728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:22:51.694469 30728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:51.694514 30728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:51.694571 30728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:22:51.694623 30728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:22:51.705332 30728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:42449
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:42449
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:51.707160 30728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:51.709369 30728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:51.724720 30737 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:51.724756 30742 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:51.724820 30739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:51.726331 30728 server_base.cc:1034] running on GCE node
I20260806 09:22:51.727473 30728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:51.729251 30728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:51.730516 30728 hybrid_clock.cc:648] HybridClock initialized: now 1786008171730450 us; error 54 us; skew 500 ppm
I20260806 09:22:51.730967 30728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:22:51.734206 30728 webserver.cc:460] Webserver started at http://127.16.86.190:40023/ using document root <none> and password file <none>
I20260806 09:22:51.735188 30728 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:51.735340 30728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:51.735762 30728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:51.739269 30728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "9008a5a39af74169afa5d920ac51af88"
format_stamp: "Formatted at 2026-08-06 09:22:51 on dist-test-slave-h3ft"
server_key: "7dbba601d528215d77900691a7f594e2"
server_key_iv: "20468bec2d5d17e085f5d6ca389f5fe6"
server_key_version: "encryptionkey@0"
I20260806 09:22:51.740214 30728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "9008a5a39af74169afa5d920ac51af88"
format_stamp: "Formatted at 2026-08-06 09:22:51 on dist-test-slave-h3ft"
server_key: "7dbba601d528215d77900691a7f594e2"
server_key_iv: "20468bec2d5d17e085f5d6ca389f5fe6"
server_key_version: "encryptionkey@0"
I20260806 09:22:51.746830 30728 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260806 09:22:51.751646 30750 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:51.753500 30728 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:22:51.753754 30728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "9008a5a39af74169afa5d920ac51af88"
format_stamp: "Formatted at 2026-08-06 09:22:51 on dist-test-slave-h3ft"
server_key: "7dbba601d528215d77900691a7f594e2"
server_key_iv: "20468bec2d5d17e085f5d6ca389f5fe6"
server_key_version: "encryptionkey@0"
I20260806 09:22:51.754024 30728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:51.760820 30728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:51.761711 30728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:51.762041 30728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:51.784886 30728 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:42449
I20260806 09:22:51.784890 30822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:42449 every 8 connection(s)
I20260806 09:22:51.786482 30728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:22:51.791589 30823 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:51.796710 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 30728
I20260806 09:22:51.796972 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:22:51.797808 16730 external_mini_cluster.cc:1428] Setting key 57918c2bff020b775dba2cbb8ddfbec8
I20260806 09:22:51.806217 30823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88: Bootstrap starting.
I20260806 09:22:51.808956 30828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:51.800505 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48912 (local address 127.16.86.190:42449)
0806 09:22:51.801942 (+ 1437us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:51.801957 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:51.802009 (+ 52us) server_negotiation.cc:407] Connection header received
0806 09:22:51.804517 (+ 2508us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:51.804548 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:51.805080 (+ 532us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:51.805533 (+ 453us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:51.806913 (+ 1380us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:51.806920 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:51.806936 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:51.806952 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:51.807088 (+ 136us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:51.807111 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:51.807396 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:51.807558 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 09:22:51.807846 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":830,"thread_start_us":500,"threads_started":1}
I20260806 09:22:51.810575 30823 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:51.812095 30823 log.cc:824] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:51.816039 30823 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88: No bootstrap required, opened a new log
I20260806 09:22:51.825049 30823 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9008a5a39af74169afa5d920ac51af88" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42449 } }
I20260806 09:22:51.825517 30823 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:51.825603 30823 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9008a5a39af74169afa5d920ac51af88, State: Initialized, Role: FOLLOWER
I20260806 09:22:51.826251 30823 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9008a5a39af74169afa5d920ac51af88" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42449 } }
I20260806 09:22:51.826488 30823 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:22:51.826601 30823 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:22:51.826756 30823 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:51.829825 30823 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9008a5a39af74169afa5d920ac51af88" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42449 } }
I20260806 09:22:51.830422 30823 leader_election.cc:302] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9008a5a39af74169afa5d920ac51af88; no voters:
I20260806 09:22:51.831208 30823 leader_election.cc:288] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:22:51.831429 30832 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:51.832363 30832 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [term 1 LEADER]: Becoming Leader. State: Replica: 9008a5a39af74169afa5d920ac51af88, State: Running, Role: LEADER
I20260806 09:22:51.833082 30832 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9008a5a39af74169afa5d920ac51af88" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42449 } }
I20260806 09:22:51.834089 30823 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:22:51.836509 30833 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9008a5a39af74169afa5d920ac51af88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9008a5a39af74169afa5d920ac51af88" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42449 } } }
I20260806 09:22:51.836890 30833 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:51.837841 30835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9008a5a39af74169afa5d920ac51af88. Latest consensus state: current_term: 1 leader_uuid: "9008a5a39af74169afa5d920ac51af88" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9008a5a39af74169afa5d920ac51af88" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42449 } } }
I20260806 09:22:51.838534 30835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88 [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:51.840018 30840 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:22:51.845993 30840 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:22:51.858290 30840 catalog_manager.cc:1347] Generated new cluster ID: d64e2670b2784ff5888c59dfaa735f96
I20260806 09:22:51.858445 30840 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:22:51.887183 30840 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:22:51.887815 30840 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:22:51.907087 30840 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9008a5a39af74169afa5d920ac51af88: Generated new TSK 0
I20260806 09:22:51.908411 30840 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:22:51.927358 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42449
--builtin_ntp_servers=127.16.86.148:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20260806 09:22:52.337551 30861 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:52.337879 30861 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:52.338021 30861 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:52.345674 30861 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:52.345795 30861 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:52.345964 30861 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:22:52.355644 30861 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:52.357084 30861 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:52.359220 30861 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:52.374230 30866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:52.374316 30869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:52.374552 30867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:52.376946 30861 server_base.cc:1034] running on GCE node
I20260806 09:22:52.377725 30861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:52.378829 30861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:52.380259 30861 hybrid_clock.cc:648] HybridClock initialized: now 1786008172380155 us; error 91 us; skew 500 ppm
I20260806 09:22:52.380671 30861 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:22:52.383616 30861 webserver.cc:460] Webserver started at http://127.16.86.129:42919/ using document root <none> and password file <none>
I20260806 09:22:52.384462 30861 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:52.384616 30861 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:52.385000 30861 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:52.388374 30861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "25297b35b03a43918691ab2f22c42949"
format_stamp: "Formatted at 2026-08-06 09:22:52 on dist-test-slave-h3ft"
server_key: "651cbd663583577f5dd0b0866bbbc89e"
server_key_iv: "a314dc7b11e06e6661f5eed2dd00e752"
server_key_version: "encryptionkey@0"
I20260806 09:22:52.389604 30861 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "25297b35b03a43918691ab2f22c42949"
format_stamp: "Formatted at 2026-08-06 09:22:52 on dist-test-slave-h3ft"
server_key: "651cbd663583577f5dd0b0866bbbc89e"
server_key_iv: "a314dc7b11e06e6661f5eed2dd00e752"
server_key_version: "encryptionkey@0"
I20260806 09:22:52.396399 30861 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 09:22:52.400740 30875 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:52.402652 30861 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:22:52.402891 30861 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "25297b35b03a43918691ab2f22c42949"
format_stamp: "Formatted at 2026-08-06 09:22:52 on dist-test-slave-h3ft"
server_key: "651cbd663583577f5dd0b0866bbbc89e"
server_key_iv: "a314dc7b11e06e6661f5eed2dd00e752"
server_key_version: "encryptionkey@0"
I20260806 09:22:52.403234 30861 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:52.410604 30861 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:52.411655 30861 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:52.411973 30861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:52.413179 30861 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:52.414896 30861 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:52.415059 30861 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:52.415215 30861 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:52.415315 30861 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:52.458469 30861 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:43657
I20260806 09:22:52.458534 31063 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:43657 every 8 connection(s)
I20260806 09:22:52.460291 30861 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:22:52.468376 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 30861
I20260806 09:22:52.468694 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:22:52.469555 16730 external_mini_cluster.cc:1428] Setting key 4f36974c1fa97d5577fa9aac4191e2b4
I20260806 09:22:52.470496 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:52.463781 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:52465 (local address 127.16.86.190:42449)
0806 09:22:52.464324 (+ 543us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:52.464329 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:52.465847 (+ 1518us) server_negotiation.cc:407] Connection header received
0806 09:22:52.467783 (+ 1936us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:52.467788 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:52.467868 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:52.467988 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:52.469590 (+ 1602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:52.469598 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:52.469602 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:52.469674 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:52.469733 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:52.469749 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:52.469893 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:52.470085 (+ 192us) server_negotiation.cc:299] Negotiation successful
0806 09:22:52.470161 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":319,"threads_started":1}
I20260806 09:22:52.471787 31067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:52.464492 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:42449 (local address 127.16.86.129:52465)
0806 09:22:52.465658 (+ 1166us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:52.465710 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:52.467496 (+ 1786us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:52.468251 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:52.468260 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:52.468350 (+ 90us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:52.469251 (+ 901us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:22:52.469254 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:52.469294 (+ 40us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:22:52.469380 (+ 86us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:52.469854 (+ 474us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:52.469867 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:52.469877 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:22:52.470191 (+ 314us) client_negotiation.cc:240] Negotiation successful
0806 09:22:52.470451 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"thread_start_us":342,"threads_started":1}
I20260806 09:22:52.473410 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42449
--builtin_ntp_servers=127.16.86.148:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 09:22:52.473738 31064 heartbeater.cc:347] Connected to a master server at 127.16.86.190:42449
I20260806 09:22:52.474076 31064 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:52.474718 31064 heartbeater.cc:511] Master 127.16.86.190:42449 requested a full tablet report, sending...
I20260806 09:22:52.477183 30776 ts_manager.cc:194] Registered new tserver with Master: 25297b35b03a43918691ab2f22c42949 (127.16.86.129:43657)
W20260806 09:22:52.837494 31068 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:52.837858 31068 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:52.838027 31068 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:52.848475 31068 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:52.848629 31068 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:52.848843 31068 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:22:52.860137 31068 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:52.861552 31068 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:52.863477 31068 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:52.878647 31074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:52.878751 31076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:52.878567 31073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:52.880062 31068 server_base.cc:1034] running on GCE node
I20260806 09:22:52.880846 31068 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:52.882153 31068 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:52.883427 31068 hybrid_clock.cc:648] HybridClock initialized: now 1786008172883356 us; error 52 us; skew 500 ppm
I20260806 09:22:52.883814 31068 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:22:52.886956 31068 webserver.cc:460] Webserver started at http://127.16.86.130:41193/ using document root <none> and password file <none>
I20260806 09:22:52.887934 31068 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:52.888089 31068 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:52.888504 31068 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:52.892283 31068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "d2c9b838ddd04d8297ddd618f0fee56c"
format_stamp: "Formatted at 2026-08-06 09:22:52 on dist-test-slave-h3ft"
server_key: "0ff7b55126b377c511bf14630d7a3519"
server_key_iv: "fc170038e61c8dfb42f20ec7765fcfbe"
server_key_version: "encryptionkey@0"
I20260806 09:22:52.893388 31068 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "d2c9b838ddd04d8297ddd618f0fee56c"
format_stamp: "Formatted at 2026-08-06 09:22:52 on dist-test-slave-h3ft"
server_key: "0ff7b55126b377c511bf14630d7a3519"
server_key_iv: "fc170038e61c8dfb42f20ec7765fcfbe"
server_key_version: "encryptionkey@0"
I20260806 09:22:52.900316 31068 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260806 09:22:52.904821 31085 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:52.906692 31068 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:22:52.906927 31068 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "d2c9b838ddd04d8297ddd618f0fee56c"
format_stamp: "Formatted at 2026-08-06 09:22:52 on dist-test-slave-h3ft"
server_key: "0ff7b55126b377c511bf14630d7a3519"
server_key_iv: "fc170038e61c8dfb42f20ec7765fcfbe"
server_key_version: "encryptionkey@0"
I20260806 09:22:52.907286 31068 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:52.914844 31068 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:52.915942 31068 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:52.916262 31068 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:52.917465 31068 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:52.919376 31068 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:52.919502 31068 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:52.919648 31068 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:52.919745 31068 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:52.962069 31068 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:46691
I20260806 09:22:52.962144 31246 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:46691 every 8 connection(s)
I20260806 09:22:52.963979 31068 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:22:52.972180 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 31068
I20260806 09:22:52.972441 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:22:52.973165 16730 external_mini_cluster.cc:1428] Setting key 25dd9f7b0c995def3b953e4927501f33
I20260806 09:22:52.976459 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42449
--builtin_ntp_servers=127.16.86.148:46691
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20260806 09:22:52.980317 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:52.967386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:37801 (local address 127.16.86.190:42449)
0806 09:22:52.967624 (+ 238us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:52.967630 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:52.969586 (+ 1956us) server_negotiation.cc:407] Connection header received
0806 09:22:52.971687 (+ 2101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:52.971693 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:52.971773 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:52.971902 (+ 129us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:52.979477 (+ 7575us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:52.979482 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:52.979485 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:52.979505 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:52.979552 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:52.979568 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:52.979726 (+ 158us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:52.979944 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 09:22:52.980024 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
I20260806 09:22:52.981448 31250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:52.968071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:42449 (local address 127.16.86.130:37801)
0806 09:22:52.969398 (+ 1327us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:52.969443 (+ 45us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:52.971367 (+ 1924us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:52.972169 (+ 802us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:52.972177 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:52.972302 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:52.978992 (+ 6690us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:22:52.979061 (+ 69us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:52.979131 (+ 70us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:22:52.979254 (+ 123us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:52.979682 (+ 428us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:52.979695 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:52.979706 (+ 11us) client_negotiation.cc:769] Sending connection context
0806 09:22:52.980043 (+ 337us) client_negotiation.cc:240] Negotiation successful
0806 09:22:52.980344 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":690,"thread_start_us":346,"threads_started":1}
I20260806 09:22:52.983734 31247 heartbeater.cc:347] Connected to a master server at 127.16.86.190:42449
I20260806 09:22:52.984169 31247 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:52.984921 31247 heartbeater.cc:511] Master 127.16.86.190:42449 requested a full tablet report, sending...
I20260806 09:22:52.986958 30776 ts_manager.cc:194] Registered new tserver with Master: d2c9b838ddd04d8297ddd618f0fee56c (127.16.86.130:46691)
W20260806 09:22:53.341574 31252 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:53.341933 31252 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:53.342113 31252 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:53.349638 31252 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:53.349792 31252 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:53.349978 31252 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:22:53.360102 31252 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46691
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42449
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:53.361498 31252 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:53.363391 31252 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:53.378296 31257 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:53.378449 31258 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:53.378367 31260 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:53.379654 31252 server_base.cc:1034] running on GCE node
I20260806 09:22:53.380618 31252 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:53.381991 31252 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:53.383277 31252 hybrid_clock.cc:648] HybridClock initialized: now 1786008173383208 us; error 51 us; skew 500 ppm
I20260806 09:22:53.383854 31252 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:22:53.387105 31252 webserver.cc:460] Webserver started at http://127.16.86.131:36061/ using document root <none> and password file <none>
I20260806 09:22:53.388043 31252 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:53.388182 31252 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:53.388610 31252 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:53.392472 31252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "336f2b2df7034b37ba9a041c403405c4"
format_stamp: "Formatted at 2026-08-06 09:22:53 on dist-test-slave-h3ft"
server_key: "fb7f780d826a48412b554c8a14719869"
server_key_iv: "43a998193cf2e306be285d4a86e58e53"
server_key_version: "encryptionkey@0"
I20260806 09:22:53.393586 31252 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "336f2b2df7034b37ba9a041c403405c4"
format_stamp: "Formatted at 2026-08-06 09:22:53 on dist-test-slave-h3ft"
server_key: "fb7f780d826a48412b554c8a14719869"
server_key_iv: "43a998193cf2e306be285d4a86e58e53"
server_key_version: "encryptionkey@0"
I20260806 09:22:53.402321 31252 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.004s
I20260806 09:22:53.407883 31271 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:53.410099 31252 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260806 09:22:53.410416 31252 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "336f2b2df7034b37ba9a041c403405c4"
format_stamp: "Formatted at 2026-08-06 09:22:53 on dist-test-slave-h3ft"
server_key: "fb7f780d826a48412b554c8a14719869"
server_key_iv: "43a998193cf2e306be285d4a86e58e53"
server_key_version: "encryptionkey@0"
I20260806 09:22:53.410717 31252 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:53.418484 31252 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:53.419705 31252 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:53.420080 31252 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:53.421379 31252 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:53.423503 31252 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:53.423643 31252 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:53.423818 31252 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:53.423911 31252 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:53.466961 31252 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:34655
I20260806 09:22:53.467085 31443 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:34655 every 8 connection(s)
I20260806 09:22:53.468766 31252 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:22:53.476114 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 31252
I20260806 09:22:53.476338 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:22:53.476949 16730 external_mini_cluster.cc:1428] Setting key d1555227a840626b017f66a03e5bb243
I20260806 09:22:53.480360 31064 heartbeater.cc:503] Master 127.16.86.190:42449 was elected leader, sending a full tablet report...
I20260806 09:22:53.480820 31066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:53.472323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:58545 (local address 127.16.86.190:42449)
0806 09:22:53.472666 (+ 343us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:53.472672 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:53.474688 (+ 2016us) server_negotiation.cc:407] Connection header received
0806 09:22:53.477308 (+ 2620us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:53.477315 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:53.477410 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:53.477568 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:53.479711 (+ 2143us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:53.479757 (+ 46us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:53.479766 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:53.479796 (+ 30us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:53.479852 (+ 56us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:53.479874 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:53.479997 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:53.480385 (+ 388us) server_negotiation.cc:299] Negotiation successful
0806 09:22:53.480462 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":115}
I20260806 09:22:53.482545 31447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:53.473156 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:42449 (local address 127.16.86.131:58545)
0806 09:22:53.474472 (+ 1316us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:53.474521 (+ 49us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:53.476847 (+ 2326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:53.477828 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:53.477837 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:53.477980 (+ 143us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:53.479197 (+ 1217us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:22:53.479200 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:53.479262 (+ 62us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:22:53.479411 (+ 149us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:53.480050 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:53.480067 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:53.480077 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:22:53.480489 (+ 412us) client_negotiation.cc:240] Negotiation successful
0806 09:22:53.480961 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":698,"thread_start_us":400,"threads_started":1}
I20260806 09:22:53.484608 31445 heartbeater.cc:347] Connected to a master server at 127.16.86.190:42449
I20260806 09:22:53.485049 31445 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:53.485798 31445 heartbeater.cc:511] Master 127.16.86.190:42449 requested a full tablet report, sending...
I20260806 09:22:53.487843 30776 ts_manager.cc:194] Registered new tserver with Master: 336f2b2df7034b37ba9a041c403405c4 (127.16.86.131:34655)
I20260806 09:22:53.494859 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
W20260806 09:22:53.502300 31066 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:22:53.500415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48916 (local address 127.16.86.190:42449)
0806 09:22:53.500711 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:53.500722 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:53.501250 (+ 528us) server_negotiation.cc:407] Connection header received
0806 09:22:53.501439 (+ 189us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:53.501443 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:53.501507 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:53.501610 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:53.502009 (+ 399us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48916: BlockingRecv error: recv got EOF from 127.0.0.1:48916 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 09:22:53.503669 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 30861
I20260806 09:22:53.508510 31458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:53.505116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55554 (local address 127.16.86.129:43657)
0806 09:22:53.505842 (+ 726us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:53.505851 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:53.505898 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:22:53.505985 (+ 87us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:53.505993 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:53.506278 (+ 285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:53.506465 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:53.507668 (+ 1203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:53.507676 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:53.507690 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:53.507702 (+ 12us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:53.507752 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:53.507767 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:53.507947 (+ 180us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:53.508003 (+ 56us) server_negotiation.cc:299] Negotiation successful
0806 09:22:53.508168 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":508,"thread_start_us":311,"threads_started":1}
I20260806 09:22:53.510118 31058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:53.597455 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 30861
I20260806 09:22:53.624563 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 31068
I20260806 09:22:53.628865 31461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:53.625754 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.16.86.130:46691)
0806 09:22:53.626331 (+ 577us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:53.626338 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:53.626370 (+ 32us) server_negotiation.cc:407] Connection header received
0806 09:22:53.626443 (+ 73us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:53.626450 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:53.626687 (+ 237us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:53.626843 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:53.628164 (+ 1321us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:53.628171 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:53.628184 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:53.628198 (+ 14us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:53.628251 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:53.628270 (+ 19us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:53.628398 (+ 128us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:53.628515 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 09:22:53.628616 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":412,"thread_start_us":274,"threads_started":1}
I20260806 09:22:53.630548 31242 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:53.718372 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 31068
I20260806 09:22:53.750013 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 31252
I20260806 09:22:53.754806 31463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:53.751290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52312 (local address 127.16.86.131:34655)
0806 09:22:53.752160 (+ 870us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:53.752169 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:53.752206 (+ 37us) server_negotiation.cc:407] Connection header received
0806 09:22:53.752283 (+ 77us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:53.752289 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:53.752557 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:53.752724 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:53.753948 (+ 1224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:53.753955 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:53.753967 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:22:53.753991 (+ 24us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:53.754041 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:22:53.754057 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:53.754224 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:53.754388 (+ 164us) server_negotiation.cc:299] Negotiation successful
0806 09:22:53.754476 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":693,"thread_start_us":516,"threads_started":1}
I20260806 09:22:53.756309 31436 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:53.857013 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 31252
I20260806 09:22:53.881494 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 30728
I20260806 09:22:53.883266 30815 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:53.992491 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 30728
2026-08-06T09:22:54Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2855 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:22:54 dist-test-slave-h3ft krb5kdc[31472](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:22:54 dist-test-slave-h3ft krb5kdc[31472](info): set up 2 sockets
Aug 06 09:22:54 dist-test-slave-h3ft krb5kdc[31472](info): commencing operation
krb5kdc: starting...
W20260806 09:22:55.371834 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.321s user 0.002s sys 0.059s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:22:55 dist-test-slave-h3ft krb5kdc[31472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:22:55Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:22:55Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:22:55.745676 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41817
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:41817
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:56.184532 31488 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:56.184867 31488 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:56.184964 31488 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:56.192334 31488 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:22:56.192454 31488 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:56.192500 31488 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:56.192642 31488 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:22:56.192724 31488 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:22:56.202827 31488 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:41817
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41817
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:56.204254 31488 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:56.205628 31488 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:56.219448 31493 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:56.219628 31497 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:56.219760 31500 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:56.221139 31488 server_base.cc:1034] running on GCE node
I20260806 09:22:56.222048 31488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:56.223654 31488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:56.224921 31488 hybrid_clock.cc:648] HybridClock initialized: now 1786008176224858 us; error 49 us; skew 500 ppm
I20260806 09:22:56.225335 31488 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:56 dist-test-slave-h3ft krb5kdc[31472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008176, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:56.228976 31488 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:22:56.231321 31488 webserver.cc:460] Webserver started at http://127.16.86.190:43625/ using document root <none> and password file <none>
I20260806 09:22:56.232112 31488 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:56.232259 31488 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:56.232607 31488 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:56.235853 31488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "b01623e1f4864093b5172ef3674345cd"
format_stamp: "Formatted at 2026-08-06 09:22:56 on dist-test-slave-h3ft"
server_key: "68dd9ca88c5b4315d31569890f1b4e92"
server_key_iv: "15c908158b9aaf92adcf8773ae063b16"
server_key_version: "encryptionkey@0"
I20260806 09:22:56.236672 31488 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "b01623e1f4864093b5172ef3674345cd"
format_stamp: "Formatted at 2026-08-06 09:22:56 on dist-test-slave-h3ft"
server_key: "68dd9ca88c5b4315d31569890f1b4e92"
server_key_iv: "15c908158b9aaf92adcf8773ae063b16"
server_key_version: "encryptionkey@0"
I20260806 09:22:56.242589 31488 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:22:56.246837 31514 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:56.248577 31488 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:22:56.248827 31488 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "b01623e1f4864093b5172ef3674345cd"
format_stamp: "Formatted at 2026-08-06 09:22:56 on dist-test-slave-h3ft"
server_key: "68dd9ca88c5b4315d31569890f1b4e92"
server_key_iv: "15c908158b9aaf92adcf8773ae063b16"
server_key_version: "encryptionkey@0"
I20260806 09:22:56.249073 31488 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:56.269552 31488 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:56.273835 31488 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:56.274143 31488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:56.295507 31488 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:41817
I20260806 09:22:56.295617 31587 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:41817 every 8 connection(s)
I20260806 09:22:56.297008 31488 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:22:56.301971 31588 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:56.305472 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 31488
I20260806 09:22:56.305828 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:22:56.306588 16730 external_mini_cluster.cc:1428] Setting key 42f7b682a671693ff93f43a3253164b8
I20260806 09:22:56.317878 31588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd: Bootstrap starting.
I20260806 09:22:56.322192 31588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:56.323597 31588 log.cc:824] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:22:56 dist-test-slave-h3ft krb5kdc[31472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:22:56.327498 31588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd: No bootstrap required, opened a new log
I20260806 09:22:56.334941 31593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:56.309463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48794 (local address 127.16.86.190:41817)
0806 09:22:56.310507 (+ 1044us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:56.310527 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:56.310599 (+ 72us) server_negotiation.cc:407] Connection header received
0806 09:22:56.312942 (+ 2343us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:56.312979 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:56.313670 (+ 691us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:56.314234 (+ 564us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:56.315991 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:56.318097 (+ 2106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:56.319762 (+ 1665us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:56.320864 (+ 1102us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:56.321086 (+ 222us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:56.325946 (+ 4860us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:56.325956 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:56.325976 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:56.326021 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:56.329837 (+ 3816us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:56.330707 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:56.330716 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:56.330724 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:56.330917 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:56.331404 (+ 487us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:56.331408 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:56.331411 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:56.332102 (+ 691us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:56.332570 (+ 468us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:56.333058 (+ 488us) server_negotiation.cc:299] Negotiation successful
0806 09:22:56.333588 (+ 530us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":430,"threads_started":1}
I20260806 09:22:56.336885 31588 raft_consensus.cc:348] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01623e1f4864093b5172ef3674345cd" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41817 } }
I20260806 09:22:56.337453 31588 raft_consensus.cc:374] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:56.337625 31588 raft_consensus.cc:729] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b01623e1f4864093b5172ef3674345cd, State: Initialized, Role: FOLLOWER
I20260806 09:22:56.338378 31588 consensus_queue.cc:260] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01623e1f4864093b5172ef3674345cd" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41817 } }
I20260806 09:22:56.338667 31588 raft_consensus.cc:388] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:22:56.338835 31588 raft_consensus.cc:482] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:22:56.339048 31588 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:56.341434 31588 raft_consensus.cc:504] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01623e1f4864093b5172ef3674345cd" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41817 } }
I20260806 09:22:56.342001 31588 leader_election.cc:302] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b01623e1f4864093b5172ef3674345cd; no voters:
I20260806 09:22:56.342689 31588 leader_election.cc:288] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:22:56.343135 31595 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:56.344223 31595 raft_consensus.cc:686] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [term 1 LEADER]: Becoming Leader. State: Replica: b01623e1f4864093b5172ef3674345cd, State: Running, Role: LEADER
I20260806 09:22:56.344892 31595 consensus_queue.cc:237] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01623e1f4864093b5172ef3674345cd" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41817 } }
I20260806 09:22:56.345139 31588 sys_catalog.cc:564] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:22:56.348925 31599 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [sys.catalog]: SysCatalogTable state changed. Reason: New leader b01623e1f4864093b5172ef3674345cd. Latest consensus state: current_term: 1 leader_uuid: "b01623e1f4864093b5172ef3674345cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01623e1f4864093b5172ef3674345cd" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41817 } } }
I20260806 09:22:56.349045 31596 sys_catalog.cc:455] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b01623e1f4864093b5172ef3674345cd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b01623e1f4864093b5172ef3674345cd" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41817 } } }
I20260806 09:22:56.349443 31599 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:56.349484 31596 sys_catalog.cc:458] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd [sys.catalog]: This master's current role is: LEADER
I20260806 09:22:56.356199 31613 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:22:56.361680 31613 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:22:56.372080 31613 catalog_manager.cc:1347] Generated new cluster ID: 6dfe0ac8b66d4e76889a044c43ec9c7c
I20260806 09:22:56.372330 31613 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:22:56.386796 31613 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:22:56.388664 31613 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:22:56.396394 31613 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd: Generated new TSK 0
I20260806 09:22:56.397596 31613 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:22:56.484160 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41817
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:56.914345 31628 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:56.914736 31628 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:56.914909 31628 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:56.923877 31628 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:56.924033 31628 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:56.924242 31628 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:22:56.935767 31628 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:56.937650 31628 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:56.939522 31628 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:56.956996 31638 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:56.957191 31637 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:56.957641 31640 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:56.958549 31628 server_base.cc:1034] running on GCE node
I20260806 09:22:56.959427 31628 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:56.960686 31628 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:56.961982 31628 hybrid_clock.cc:648] HybridClock initialized: now 1786008176961909 us; error 53 us; skew 500 ppm
I20260806 09:22:56.962414 31628 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:56 dist-test-slave-h3ft krb5kdc[31472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008176, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:56.966149 31628 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:22:56.968014 31628 webserver.cc:460] Webserver started at http://127.16.86.129:41169/ using document root <none> and password file <none>
I20260806 09:22:56.968787 31628 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:56.968897 31628 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:56.969190 31628 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:56.972319 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "61c5e3137c3947919e5cfc545e5ada03"
format_stamp: "Formatted at 2026-08-06 09:22:56 on dist-test-slave-h3ft"
server_key: "97e3022929be12ce10fbc4b1cdad9f07"
server_key_iv: "6f6ccc121da87172bc2f767257a35b0e"
server_key_version: "encryptionkey@0"
I20260806 09:22:56.973093 31628 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "61c5e3137c3947919e5cfc545e5ada03"
format_stamp: "Formatted at 2026-08-06 09:22:56 on dist-test-slave-h3ft"
server_key: "97e3022929be12ce10fbc4b1cdad9f07"
server_key_iv: "6f6ccc121da87172bc2f767257a35b0e"
server_key_version: "encryptionkey@0"
I20260806 09:22:56.979135 31628 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260806 09:22:56.983294 31648 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:56.985006 31628 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260806 09:22:56.985250 31628 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "61c5e3137c3947919e5cfc545e5ada03"
format_stamp: "Formatted at 2026-08-06 09:22:56 on dist-test-slave-h3ft"
server_key: "97e3022929be12ce10fbc4b1cdad9f07"
server_key_iv: "6f6ccc121da87172bc2f767257a35b0e"
server_key_version: "encryptionkey@0"
I20260806 09:22:56.985445 31628 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:57.007297 31628 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:57.012184 31628 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:57.012519 31628 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:57.013707 31628 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:57.015676 31628 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:57.015764 31628 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:57.015946 31628 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:57.016014 31628 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:57.057624 31628 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:39223
I20260806 09:22:57.057668 31811 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:39223 every 8 connection(s)
I20260806 09:22:57.059118 31628 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:22:57.066505 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 31628
I20260806 09:22:57.066813 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:22:57.067754 16730 external_mini_cluster.cc:1428] Setting key bdc92803039438e43ad1ee9be787b52d
Aug 06 09:22:57 dist-test-slave-h3ft krb5kdc[31472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008176, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:22:57.088940 31815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:57.062771 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:59641 (local address 127.16.86.190:41817)
0806 09:22:57.063518 (+ 747us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:57.063524 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:57.064873 (+ 1349us) server_negotiation.cc:407] Connection header received
0806 09:22:57.068035 (+ 3162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:57.068041 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:57.068172 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:57.068317 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:57.070783 (+ 2466us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:57.071666 (+ 883us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:57.074766 (+ 3100us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:57.075310 (+ 544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:57.075452 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:57.081198 (+ 5746us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:57.081204 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:57.081207 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:57.081252 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:57.084347 (+ 3095us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:57.085689 (+ 1342us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:57.085694 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:57.085696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:57.085853 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:57.086477 (+ 624us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:57.086481 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:57.086483 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:57.086862 (+ 379us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:57.086989 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:57.088160 (+ 1171us) server_negotiation.cc:299] Negotiation successful
0806 09:22:57.088449 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":584,"thread_start_us":422,"threads_started":1}
I20260806 09:22:57.090312 31816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:57.063504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41817 (local address 127.16.86.129:59641)
0806 09:22:57.064640 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:57.064695 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:57.067516 (+ 2821us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:57.068703 (+ 1187us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:57.068715 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:57.069590 (+ 875us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:57.070546 (+ 956us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:57.070576 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:57.071875 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:57.071883 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:57.074495 (+ 2612us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:57.074520 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:57.075635 (+ 1115us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:57.075645 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:57.076060 (+ 415us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:57.077101 (+ 1041us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:57.077152 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:57.080836 (+ 3684us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:57.084567 (+ 3731us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:57.084584 (+ 17us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:57.084623 (+ 39us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:57.085497 (+ 874us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:57.086004 (+ 507us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:57.086011 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:57.086017 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:57.086325 (+ 308us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:57.087051 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:57.087063 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:57.087692 (+ 629us) client_negotiation.cc:769] Sending connection context
0806 09:22:57.088004 (+ 312us) client_negotiation.cc:240] Negotiation successful
0806 09:22:57.088553 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":792,"thread_start_us":522,"threads_started":1}
I20260806 09:22:57.092521 31812 heartbeater.cc:347] Connected to a master server at 127.16.86.190:41817
I20260806 09:22:57.093014 31812 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:57.094082 31812 heartbeater.cc:511] Master 127.16.86.190:41817 requested a full tablet report, sending...
I20260806 09:22:57.096907 31537 ts_manager.cc:194] Registered new tserver with Master: 61c5e3137c3947919e5cfc545e5ada03 (127.16.86.129:39223)
I20260806 09:22:57.098740 31537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:59641
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:22:57.151639 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41817
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:22:57.606027 31819 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:57.606511 31819 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:57.606766 31819 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:57.615247 31819 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:57.615417 31819 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:57.615553 31819 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:22:57.626768 31819 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:57.628616 31819 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:57.630326 31819 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:57.646952 31826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:57.647076 31828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:57.647137 31830 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:57.648146 31819 server_base.cc:1034] running on GCE node
I20260806 09:22:57.648991 31819 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:57.650381 31819 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:57.651739 31819 hybrid_clock.cc:648] HybridClock initialized: now 1786008177651676 us; error 53 us; skew 500 ppm
I20260806 09:22:57.652168 31819 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:57 dist-test-slave-h3ft krb5kdc[31472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008177, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:57.655892 31819 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:22:57.657747 31819 webserver.cc:460] Webserver started at http://127.16.86.130:33989/ using document root <none> and password file <none>
I20260806 09:22:57.658618 31819 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:57.658733 31819 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:57.659093 31819 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:57.662279 31819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "db7c24c76851428a9fa52e91d27560b8"
format_stamp: "Formatted at 2026-08-06 09:22:57 on dist-test-slave-h3ft"
server_key: "847ba0ba03adced755f67eeb5dfe6c77"
server_key_iv: "39ef39d7c5482f31f898605d1998666c"
server_key_version: "encryptionkey@0"
I20260806 09:22:57.663120 31819 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "db7c24c76851428a9fa52e91d27560b8"
format_stamp: "Formatted at 2026-08-06 09:22:57 on dist-test-slave-h3ft"
server_key: "847ba0ba03adced755f67eeb5dfe6c77"
server_key_iv: "39ef39d7c5482f31f898605d1998666c"
server_key_version: "encryptionkey@0"
I20260806 09:22:57.669227 31819 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260806 09:22:57.673377 31839 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:57.675297 31819 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:22:57.675523 31819 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "db7c24c76851428a9fa52e91d27560b8"
format_stamp: "Formatted at 2026-08-06 09:22:57 on dist-test-slave-h3ft"
server_key: "847ba0ba03adced755f67eeb5dfe6c77"
server_key_iv: "39ef39d7c5482f31f898605d1998666c"
server_key_version: "encryptionkey@0"
I20260806 09:22:57.675781 31819 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:57.696600 31819 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:57.701810 31819 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:57.702198 31819 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:57.703562 31819 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:57.705312 31819 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:57.705406 31819 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:57.705560 31819 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:57.705623 31819 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:57.747584 31819 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:43805
I20260806 09:22:57.747705 31999 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:43805 every 8 connection(s)
I20260806 09:22:57.749174 31819 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:22:57.750329 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 31819
I20260806 09:22:57.750548 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:22:57.751065 16730 external_mini_cluster.cc:1428] Setting key ae518a902987e4fd7fdc54c177d4465d
Aug 06 09:22:57 dist-test-slave-h3ft krb5kdc[31472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008177, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:22:57.775966 32003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:57.752910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:52017 (local address 127.16.86.190:41817)
0806 09:22:57.753531 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:57.753539 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:57.755064 (+ 1525us) server_negotiation.cc:407] Connection header received
0806 09:22:57.757262 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:57.757268 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:57.757371 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:57.757515 (+ 144us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:57.759603 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:57.760466 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:57.762197 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:57.762650 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:57.762792 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:57.767534 (+ 4742us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:57.767541 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:57.767545 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:57.767586 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:57.771664 (+ 4078us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:57.772826 (+ 1162us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:57.772834 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:57.772841 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:57.773021 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:57.773511 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:57.773518 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:57.773524 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:57.774038 (+ 514us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:57.774229 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:57.775114 (+ 885us) server_negotiation.cc:299] Negotiation successful
0806 09:22:57.775429 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":456,"thread_start_us":295,"threads_started":1}
I20260806 09:22:57.776722 32004 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:57.753810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41817 (local address 127.16.86.130:52017)
0806 09:22:57.754825 (+ 1015us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:57.754885 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:57.756930 (+ 2045us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:57.757856 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:57.757866 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:57.758575 (+ 709us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:57.759410 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:57.759430 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:57.760659 (+ 1229us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:57.760664 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:57.762043 (+ 1379us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:57.762056 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:57.762929 (+ 873us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:57.762935 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:57.763312 (+ 377us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:57.764238 (+ 926us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:57.764270 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:57.767238 (+ 2968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:57.771910 (+ 4672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:57.771921 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:57.771938 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:57.772641 (+ 703us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:57.773159 (+ 518us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:57.773162 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:57.773165 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:57.773372 (+ 207us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:57.774207 (+ 835us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:57.774214 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:57.774608 (+ 394us) client_negotiation.cc:769] Sending connection context
0806 09:22:57.774870 (+ 262us) client_negotiation.cc:240] Negotiation successful
0806 09:22:57.775422 (+ 552us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":677,"thread_start_us":420,"threads_started":1}
I20260806 09:22:57.778638 32001 heartbeater.cc:347] Connected to a master server at 127.16.86.190:41817
I20260806 09:22:57.779141 32001 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:57.780395 32001 heartbeater.cc:511] Master 127.16.86.190:41817 requested a full tablet report, sending...
I20260806 09:22:57.782649 31537 ts_manager.cc:194] Registered new tserver with Master: db7c24c76851428a9fa52e91d27560b8 (127.16.86.130:43805)
I20260806 09:22:57.783654 31537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:52017
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:22:57.838516 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41817
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33119
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:22:58.102813 31812 heartbeater.cc:503] Master 127.16.86.190:41817 was elected leader, sending a full tablet report...
W20260806 09:22:58.221524 32007 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:22:58.221907 32007 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:22:58.222095 32007 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:22:58.229640 32007 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:22:58.229806 32007 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:22:58.229990 32007 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:22:58.240203 32007 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33119
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:41817
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:22:58.241798 32007 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:22:58.243315 32007 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:22:58.259366 32014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:58.259599 32013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:22:58.259661 32016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:22:58.260924 32007 server_base.cc:1034] running on GCE node
I20260806 09:22:58.261701 32007 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:22:58.262799 32007 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:22:58.264076 32007 hybrid_clock.cc:648] HybridClock initialized: now 1786008178264027 us; error 98 us; skew 500 ppm
I20260806 09:22:58.264509 32007 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:22:58 dist-test-slave-h3ft krb5kdc[31472](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008178, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:22:58.274168 32007 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:22:58.276055 32007 webserver.cc:460] Webserver started at http://127.16.86.131:39559/ using document root <none> and password file <none>
I20260806 09:22:58.276854 32007 fs_manager.cc:359] Metadata directory not provided
I20260806 09:22:58.277014 32007 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:22:58.277374 32007 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:22:58.280570 32007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "fd9824357c5a44bfb6681efdea723cc8"
format_stamp: "Formatted at 2026-08-06 09:22:58 on dist-test-slave-h3ft"
server_key: "a8f33b5c77cfba85a8cc2bd22a5142b4"
server_key_iv: "7b66a256aab35a00eaaedf1a60b0c7be"
server_key_version: "encryptionkey@0"
I20260806 09:22:58.281509 32007 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "fd9824357c5a44bfb6681efdea723cc8"
format_stamp: "Formatted at 2026-08-06 09:22:58 on dist-test-slave-h3ft"
server_key: "a8f33b5c77cfba85a8cc2bd22a5142b4"
server_key_iv: "7b66a256aab35a00eaaedf1a60b0c7be"
server_key_version: "encryptionkey@0"
I20260806 09:22:58.287321 32007 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 09:22:58.291558 32024 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:58.293308 32007 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:22:58.293535 32007 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "fd9824357c5a44bfb6681efdea723cc8"
format_stamp: "Formatted at 2026-08-06 09:22:58 on dist-test-slave-h3ft"
server_key: "a8f33b5c77cfba85a8cc2bd22a5142b4"
server_key_iv: "7b66a256aab35a00eaaedf1a60b0c7be"
server_key_version: "encryptionkey@0"
I20260806 09:22:58.293790 32007 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:22:58.317929 32007 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:22:58.322669 32007 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:22:58.323024 32007 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:22:58.324144 32007 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:22:58.325663 32007 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:22:58.325783 32007 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:58.325948 32007 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:22:58.326027 32007 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:22:58.367509 32007 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:40875
I20260806 09:22:58.367617 32197 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:40875 every 8 connection(s)
I20260806 09:22:58.368916 32007 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:22:58.377472 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 32007
I20260806 09:22:58.377743 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:22:58.378333 16730 external_mini_cluster.cc:1428] Setting key 82d911765de590af82e601f8007b689e
Aug 06 09:22:58 dist-test-slave-h3ft krb5kdc[31472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008178, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:22:58.392757 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.372297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:42093 (local address 127.16.86.190:41817)
0806 09:22:58.372868 (+ 571us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:58.372874 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:58.374082 (+ 1208us) server_negotiation.cc:407] Connection header received
0806 09:22:58.376159 (+ 2077us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:58.376166 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:58.376282 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:58.376413 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:58.378189 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.379144 (+ 955us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.380947 (+ 1803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.381461 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.381602 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.385638 (+ 4036us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:58.385644 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:58.385648 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:58.385695 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:58.388853 (+ 3158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:58.389942 (+ 1089us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:58.389947 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:58.389950 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:58.390108 (+ 158us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:58.390658 (+ 550us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:58.390662 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:58.390664 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:58.390983 (+ 319us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:58.391250 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:58.392106 (+ 856us) server_negotiation.cc:299] Negotiation successful
0806 09:22:58.392326 (+ 220us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":284,"threads_started":1}
I20260806 09:22:58.394382 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.373003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41817 (local address 127.16.86.131:42093)
0806 09:22:58.373881 (+ 878us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:58.373939 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:58.375856 (+ 1917us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:58.376669 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:58.376679 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:58.377285 (+ 606us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:22:58.378013 (+ 728us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:58.378032 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.379297 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.379307 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:58.380764 (+ 1457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:58.380779 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.381703 (+ 924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.381711 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:58.381945 (+ 234us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.382689 (+ 744us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:22:58.382713 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:22:58.385352 (+ 2639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:22:58.389061 (+ 3709us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:58.389070 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:58.389088 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:58.389803 (+ 715us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:58.390229 (+ 426us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:22:58.390233 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:22:58.390235 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:22:58.390452 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:22:58.391218 (+ 766us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:22:58.391225 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:22:58.391681 (+ 456us) client_negotiation.cc:769] Sending connection context
0806 09:22:58.391985 (+ 304us) client_negotiation.cc:240] Negotiation successful
0806 09:22:58.392734 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":583,"thread_start_us":323,"threads_started":1}
I20260806 09:22:58.396191 32198 heartbeater.cc:347] Connected to a master server at 127.16.86.190:41817
I20260806 09:22:58.396591 32198 heartbeater.cc:464] Registering TS with master...
I20260806 09:22:58.397600 32198 heartbeater.cc:511] Master 127.16.86.190:41817 requested a full tablet report, sending...
I20260806 09:22:58.399627 31537 ts_manager.cc:194] Registered new tserver with Master: fd9824357c5a44bfb6681efdea723cc8 (127.16.86.131:40875)
I20260806 09:22:58.400619 31537 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:42093
I20260806 09:22:58.408315 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:22:58.428756 32200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.413971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48802 (local address 127.16.86.190:41817)
0806 09:22:58.414294 (+ 323us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:58.414305 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:58.414856 (+ 551us) server_negotiation.cc:407] Connection header received
0806 09:22:58.415088 (+ 232us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:58.415093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:58.415194 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:58.415302 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:58.417037 (+ 1735us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.417872 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.419421 (+ 1549us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.419873 (+ 452us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.419991 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.422647 (+ 2656us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:58.422652 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:58.422656 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:58.422692 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:58.425744 (+ 3052us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:58.426495 (+ 751us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:58.426499 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:58.426502 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:58.426675 (+ 173us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:58.427091 (+ 416us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:58.427095 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:58.427097 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:58.427494 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:58.427761 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:58.428117 (+ 356us) server_negotiation.cc:299] Negotiation successful
0806 09:22:58.428342 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 09:22:58.434044 31537 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48802:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:22:58.442119 31537 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:22:58.472244 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.458604 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60562 (local address 127.16.86.129:39223)
0806 09:22:58.459596 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:58.459614 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:58.460448 (+ 834us) server_negotiation.cc:407] Connection header received
0806 09:22:58.460976 (+ 528us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:58.460990 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:58.461360 (+ 370us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:58.461656 (+ 296us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:58.463364 (+ 1708us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.465666 (+ 2302us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.468607 (+ 2941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.470816 (+ 2209us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.470982 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.471337 (+ 355us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:58.471640 (+ 303us) server_negotiation.cc:299] Negotiation successful
0806 09:22:58.471824 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":724,"thread_start_us":773,"threads_started":1}
I20260806 09:22:58.472420 32215 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.458514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39223 (local address 127.0.0.1:60562)
0806 09:22:58.460280 (+ 1766us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:58.460336 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:58.460640 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:58.462075 (+ 1435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:58.462088 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:58.462136 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:58.462955 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:58.462977 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.465891 (+ 2914us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.465901 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:58.468407 (+ 2506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:58.468422 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.471088 (+ 2666us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.471097 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:58.471327 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.471387 (+ 60us) client_negotiation.cc:769] Sending connection context
0806 09:22:58.471564 (+ 177us) client_negotiation.cc:240] Negotiation successful
0806 09:22:58.471816 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1493,"mutex_wait_us":73,"thread_start_us":1344,"threads_started":1}
I20260806 09:22:58.473244 32219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.458922 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:40875 (local address 127.0.0.1:43426)
0806 09:22:58.461266 (+ 2344us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:58.461284 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:58.461421 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:58.462556 (+ 1135us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:58.462563 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:58.462612 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:58.463383 (+ 771us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:58.463403 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.465876 (+ 2473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.465882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:58.469819 (+ 3937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:58.469832 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.472251 (+ 2419us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.472261 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:58.472474 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.472521 (+ 47us) client_negotiation.cc:769] Sending connection context
0806 09:22:58.472653 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 09:22:58.472819 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2224,"thread_start_us":11676,"threads_started":1}
I20260806 09:22:58.474236 32218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.459826 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43426 (local address 127.16.86.131:40875)
0806 09:22:58.460627 (+ 801us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:58.460639 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:58.461585 (+ 946us) server_negotiation.cc:407] Connection header received
0806 09:22:58.461713 (+ 128us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:58.461722 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:58.461997 (+ 275us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:58.462191 (+ 194us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:58.463672 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.465679 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.470007 (+ 4328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.471622 (+ 1615us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.471764 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.472028 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:58.473665 (+ 1637us) server_negotiation.cc:299] Negotiation successful
0806 09:22:58.473881 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":411,"threads_started":1}
I20260806 09:22:58.477583 31718 tablet_service.cc:1467] Processing CreateTablet for tablet 84645160c0ff4e0894160e303e162bb7 (DEFAULT_TABLE table=test-table [id=27f3095b5e4b4e8f81bdffece5efdc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:58.479235 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.460711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:43805 (local address 127.0.0.1:42144)
0806 09:22:58.470891 (+ 10180us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:58.470908 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:58.471071 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:58.472113 (+ 1042us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:58.472123 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:58.472155 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:58.472629 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:58.472644 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.474718 (+ 2074us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.474723 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:58.476346 (+ 1623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:58.476359 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.478418 (+ 2059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.478424 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:58.478593 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.478626 (+ 33us) client_negotiation.cc:769] Sending connection context
0806 09:22:58.478737 (+ 111us) client_negotiation.cc:240] Negotiation successful
0806 09:22:58.478870 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10049,"thread_start_us":1910,"threads_started":1}
I20260806 09:22:58.479455 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.461266 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42144 (local address 127.16.86.130:43805)
0806 09:22:58.467733 (+ 6467us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:58.467746 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:58.471228 (+ 3482us) server_negotiation.cc:407] Connection header received
0806 09:22:58.471355 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:58.471363 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:58.471640 (+ 277us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:58.471841 (+ 201us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:58.472817 (+ 976us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.474546 (+ 1729us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.476561 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.478162 (+ 1601us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.478339 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.478580 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:58.478819 (+ 239us) server_negotiation.cc:299] Negotiation successful
0806 09:22:58.479051 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6207,"thread_start_us":6012,"threads_started":1}
I20260806 09:22:58.479768 31718 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84645160c0ff4e0894160e303e162bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:58.480464 32094 tablet_service.cc:1467] Processing CreateTablet for tablet 84645160c0ff4e0894160e303e162bb7 (DEFAULT_TABLE table=test-table [id=27f3095b5e4b4e8f81bdffece5efdc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:58.482894 32094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84645160c0ff4e0894160e303e162bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:58.483706 31907 tablet_service.cc:1467] Processing CreateTablet for tablet 84645160c0ff4e0894160e303e162bb7 (DEFAULT_TABLE table=test-table [id=27f3095b5e4b4e8f81bdffece5efdc55]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:22:58.485056 31907 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 84645160c0ff4e0894160e303e162bb7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:22:58.493960 32223 tablet_bootstrap.cc:492] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03: Bootstrap starting.
I20260806 09:22:58.498560 32223 tablet_bootstrap.cc:654] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:58.499903 32225 tablet_bootstrap.cc:492] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8: Bootstrap starting.
I20260806 09:22:58.500581 32223 log.cc:824] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:58.501955 32226 tablet_bootstrap.cc:492] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8: Bootstrap starting.
I20260806 09:22:58.504484 32225 tablet_bootstrap.cc:654] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:58.505530 32226 tablet_bootstrap.cc:654] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8: Neither blocks nor log segments found. Creating new log.
I20260806 09:22:58.505973 32225 log.cc:824] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:58.506199 32223 tablet_bootstrap.cc:492] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03: No bootstrap required, opened a new log
I20260806 09:22:58.506560 32223 ts_tablet_manager.cc:1397] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260806 09:22:58.506809 32226 log.cc:824] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8: Log is configured to *not* fsync() on all Append() calls
I20260806 09:22:58.509733 32225 tablet_bootstrap.cc:492] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8: No bootstrap required, opened a new log
I20260806 09:22:58.510083 32225 ts_tablet_manager.cc:1397] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8: Time spent bootstrapping tablet: real 0.010s user 0.008s sys 0.000s
I20260806 09:22:58.510807 32226 tablet_bootstrap.cc:492] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8: No bootstrap required, opened a new log
I20260806 09:22:58.511175 32226 ts_tablet_manager.cc:1397] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 09:22:58.517393 32225 raft_consensus.cc:348] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5e3137c3947919e5cfc545e5ada03" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39223 } } peers { permanent_uuid: "db7c24c76851428a9fa52e91d27560b8" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 43805 } } peers { permanent_uuid: "fd9824357c5a44bfb6681efdea723cc8" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 40875 } }
I20260806 09:22:58.517844 32225 raft_consensus.cc:374] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:58.517939 32225 raft_consensus.cc:729] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd9824357c5a44bfb6681efdea723cc8, State: Initialized, Role: FOLLOWER
I20260806 09:22:58.518604 32225 consensus_queue.cc:260] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5e3137c3947919e5cfc545e5ada03" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39223 } } peers { permanent_uuid: "db7c24c76851428a9fa52e91d27560b8" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 43805 } } peers { permanent_uuid: "fd9824357c5a44bfb6681efdea723cc8" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 40875 } }
I20260806 09:22:58.518602 32223 raft_consensus.cc:348] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5e3137c3947919e5cfc545e5ada03" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39223 } } peers { permanent_uuid: "db7c24c76851428a9fa52e91d27560b8" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 43805 } } peers { permanent_uuid: "fd9824357c5a44bfb6681efdea723cc8" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 40875 } }
I20260806 09:22:58.519184 32223 raft_consensus.cc:374] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:58.519307 32223 raft_consensus.cc:729] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 61c5e3137c3947919e5cfc545e5ada03, State: Initialized, Role: FOLLOWER
I20260806 09:22:58.519965 32198 heartbeater.cc:503] Master 127.16.86.190:41817 was elected leader, sending a full tablet report...
I20260806 09:22:58.520164 32223 consensus_queue.cc:260] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5e3137c3947919e5cfc545e5ada03" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39223 } } peers { permanent_uuid: "db7c24c76851428a9fa52e91d27560b8" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 43805 } } peers { permanent_uuid: "fd9824357c5a44bfb6681efdea723cc8" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 40875 } }
I20260806 09:22:58.521514 32225 ts_tablet_manager.cc:1428] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8: Time spent starting tablet: real 0.011s user 0.007s sys 0.004s
I20260806 09:22:58.523888 32230 raft_consensus.cc:482] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:22:58.524219 32230 raft_consensus.cc:504] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5e3137c3947919e5cfc545e5ada03" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39223 } } peers { permanent_uuid: "db7c24c76851428a9fa52e91d27560b8" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 43805 } } peers { permanent_uuid: "fd9824357c5a44bfb6681efdea723cc8" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 40875 } }
I20260806 09:22:58.524199 32226 raft_consensus.cc:348] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5e3137c3947919e5cfc545e5ada03" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39223 } } peers { permanent_uuid: "db7c24c76851428a9fa52e91d27560b8" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 43805 } } peers { permanent_uuid: "fd9824357c5a44bfb6681efdea723cc8" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 40875 } }
I20260806 09:22:58.524874 32226 raft_consensus.cc:374] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:22:58.525025 32226 raft_consensus.cc:729] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: db7c24c76851428a9fa52e91d27560b8, State: Initialized, Role: FOLLOWER
I20260806 09:22:58.525717 32226 consensus_queue.cc:260] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5e3137c3947919e5cfc545e5ada03" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39223 } } peers { permanent_uuid: "db7c24c76851428a9fa52e91d27560b8" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 43805 } } peers { permanent_uuid: "fd9824357c5a44bfb6681efdea723cc8" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 40875 } }
I20260806 09:22:58.526670 32230 leader_election.cc:288] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 61c5e3137c3947919e5cfc545e5ada03 (127.16.86.129:39223), db7c24c76851428a9fa52e91d27560b8 (127.16.86.130:43805)
I20260806 09:22:58.527657 32223 ts_tablet_manager.cc:1428] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03: Time spent starting tablet: real 0.021s user 0.015s sys 0.005s
I20260806 09:22:58.528909 32001 heartbeater.cc:503] Master 127.16.86.190:41817 was elected leader, sending a full tablet report...
I20260806 09:22:58.529560 32226 ts_tablet_manager.cc:1428] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8: Time spent starting tablet: real 0.018s user 0.009s sys 0.007s
I20260806 09:22:58.535701 32201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.527526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39223 (local address 127.16.86.131:39787)
0806 09:22:58.528045 (+ 519us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:58.528074 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:58.528242 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:58.529013 (+ 771us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:58.529020 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:58.529060 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:58.529292 (+ 232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:58.529311 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.531192 (+ 1881us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.531198 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:58.533185 (+ 1987us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:58.533204 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.534982 (+ 1778us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.534988 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:58.535257 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.535286 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 09:22:58.535369 (+ 83us) client_negotiation.cc:240] Negotiation successful
0806 09:22:58.535475 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":308}
I20260806 09:22:58.535872 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.527751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:39787 (local address 127.16.86.129:39223)
0806 09:22:58.528515 (+ 764us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:58.528521 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:58.528540 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:22:58.528621 (+ 81us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:58.528630 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:58.528715 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:58.528811 (+ 96us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:58.529500 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.530704 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.533391 (+ 2687us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.534787 (+ 1396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.534974 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.535135 (+ 161us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:58.535430 (+ 295us) server_negotiation.cc:299] Negotiation successful
0806 09:22:58.535589 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611}
I20260806 09:22:58.537000 31742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84645160c0ff4e0894160e303e162bb7" candidate_uuid: "fd9824357c5a44bfb6681efdea723cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c5e3137c3947919e5cfc545e5ada03" is_pre_election: true
I20260806 09:22:58.537679 31742 raft_consensus.cc:2454] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd9824357c5a44bfb6681efdea723cc8 in term 0.
I20260806 09:22:58.538488 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.528625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:51699 (local address 127.16.86.130:43805)
0806 09:22:58.528844 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:58.528851 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:58.531262 (+ 2411us) server_negotiation.cc:407] Connection header received
0806 09:22:58.531561 (+ 299us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:58.531566 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:58.531647 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:58.531715 (+ 68us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:22:58.532732 (+ 1017us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.533731 (+ 999us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.536213 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.537468 (+ 1255us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.537659 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.537748 (+ 89us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:58.538015 (+ 267us) server_negotiation.cc:299] Negotiation successful
0806 09:22:58.538198 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:22:58.538511 32026 leader_election.cc:302] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61c5e3137c3947919e5cfc545e5ada03, fd9824357c5a44bfb6681efdea723cc8; no voters:
I20260806 09:22:58.538865 32234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.527999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:43805 (local address 127.16.86.131:51699)
0806 09:22:58.531145 (+ 3146us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:58.531168 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:22:58.531379 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:22:58.531950 (+ 571us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:22:58.531955 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:22:58.531979 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:22:58.532485 (+ 506us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:58.532504 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.533906 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.533912 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:58.536042 (+ 2130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:22:58.536058 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.537680 (+ 1622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.537685 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:22:58.537811 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.537838 (+ 27us) client_negotiation.cc:769] Sending connection context
0806 09:22:58.537971 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 09:22:58.538345 (+ 374us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2948,"thread_start_us":2701,"threads_started":1}
I20260806 09:22:58.539081 32230 raft_consensus.cc:2781] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:22:58.539266 32230 raft_consensus.cc:482] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:22:58.539399 32230 raft_consensus.cc:3037] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:58.540117 31934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84645160c0ff4e0894160e303e162bb7" candidate_uuid: "fd9824357c5a44bfb6681efdea723cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7c24c76851428a9fa52e91d27560b8" is_pre_election: true
I20260806 09:22:58.540618 31934 raft_consensus.cc:2454] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fd9824357c5a44bfb6681efdea723cc8 in term 0.
I20260806 09:22:58.541970 32230 raft_consensus.cc:504] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5e3137c3947919e5cfc545e5ada03" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39223 } } peers { permanent_uuid: "db7c24c76851428a9fa52e91d27560b8" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 43805 } } peers { permanent_uuid: "fd9824357c5a44bfb6681efdea723cc8" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 40875 } }
I20260806 09:22:58.542523 32230 leader_election.cc:288] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [CANDIDATE]: Term 1 election: Requested vote from peers 61c5e3137c3947919e5cfc545e5ada03 (127.16.86.129:39223), db7c24c76851428a9fa52e91d27560b8 (127.16.86.130:43805)
I20260806 09:22:58.543263 31934 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84645160c0ff4e0894160e303e162bb7" candidate_uuid: "fd9824357c5a44bfb6681efdea723cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "db7c24c76851428a9fa52e91d27560b8"
I20260806 09:22:58.543259 31742 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "84645160c0ff4e0894160e303e162bb7" candidate_uuid: "fd9824357c5a44bfb6681efdea723cc8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "61c5e3137c3947919e5cfc545e5ada03"
I20260806 09:22:58.543506 31934 raft_consensus.cc:3037] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:58.543506 31742 raft_consensus.cc:3037] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:22:58.547325 31742 raft_consensus.cc:2454] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd9824357c5a44bfb6681efdea723cc8 in term 1.
I20260806 09:22:58.547325 31934 raft_consensus.cc:2454] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fd9824357c5a44bfb6681efdea723cc8 in term 1.
I20260806 09:22:58.547890 32026 leader_election.cc:302] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 61c5e3137c3947919e5cfc545e5ada03, fd9824357c5a44bfb6681efdea723cc8; no voters:
I20260806 09:22:58.548441 32230 raft_consensus.cc:2781] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:22:58.549237 32230 raft_consensus.cc:686] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 1 LEADER]: Becoming Leader. State: Replica: fd9824357c5a44bfb6681efdea723cc8, State: Running, Role: LEADER
I20260806 09:22:58.549888 32230 consensus_queue.cc:237] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5e3137c3947919e5cfc545e5ada03" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39223 } } peers { permanent_uuid: "db7c24c76851428a9fa52e91d27560b8" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 43805 } } peers { permanent_uuid: "fd9824357c5a44bfb6681efdea723cc8" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 40875 } }
I20260806 09:22:58.555678 31537 catalog_manager.cc:5495] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 reported cstate change: term changed from 0 to 1, leader changed from <none> to fd9824357c5a44bfb6681efdea723cc8 (127.16.86.131). New cstate: current_term: 1 leader_uuid: "fd9824357c5a44bfb6681efdea723cc8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "61c5e3137c3947919e5cfc545e5ada03" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39223 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "db7c24c76851428a9fa52e91d27560b8" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 43805 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fd9824357c5a44bfb6681efdea723cc8" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 40875 } health_report { overall_health: HEALTHY } } }
W20260806 09:22:58.562409 31814 tablet.cc:2369] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:22:58.600078 32218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.593895 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45336 (local address 127.16.86.131:40875)
0806 09:22:58.594127 (+ 232us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:58.594132 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:58.594154 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:22:58.594281 (+ 127us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:58.594285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:58.594385 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:58.594568 (+ 183us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:22:58.595065 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.595905 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.597332 (+ 1427us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.597888 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.598009 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.598582 (+ 573us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:22:58.599437 (+ 855us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:22:58.599587 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:58.599733 (+ 146us) server_negotiation.cc:299] Negotiation successful
0806 09:22:58.599815 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:22:58.611107 31934 raft_consensus.cc:1261] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8 [term 1 FOLLOWER]: Refusing update from remote peer fd9824357c5a44bfb6681efdea723cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:58.611107 31742 raft_consensus.cc:1261] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03 [term 1 FOLLOWER]: Refusing update from remote peer fd9824357c5a44bfb6681efdea723cc8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:22:58.612533 32235 consensus_queue.cc:1035] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "db7c24c76851428a9fa52e91d27560b8" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 43805 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:22:58.613071 32230 consensus_queue.cc:1035] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "61c5e3137c3947919e5cfc545e5ada03" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39223 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 09:22:58.620721 32199 tablet.cc:2369] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:22:58.645159 31536 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:58.645526 31536 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48802:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:22:58.650310 31536 catalog_manager.cc:5782] T 00000000000000000000000000000000 P b01623e1f4864093b5172ef3674345cd: Sending DeleteTablet for 3 replicas of tablet 84645160c0ff4e0894160e303e162bb7
I20260806 09:22:58.652051 32094 tablet_service.cc:1514] Processing DeleteTablet for tablet 84645160c0ff4e0894160e303e162bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:58 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:43426
I20260806 09:22:58.652051 31718 tablet_service.cc:1514] Processing DeleteTablet for tablet 84645160c0ff4e0894160e303e162bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:58 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:60562
I20260806 09:22:58.652227 31907 tablet_service.cc:1514] Processing DeleteTablet for tablet 84645160c0ff4e0894160e303e162bb7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:22:58 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:42144
I20260806 09:22:58.655730 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 31628
I20260806 09:22:58.663753 32261 tablet_replica.cc:330] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8: stopping tablet replica
I20260806 09:22:58.664862 32261 raft_consensus.cc:2229] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:22:58.666085 32261 raft_consensus.cc:2258] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:58.669461 32261 ts_tablet_manager.cc:1905] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:58.670486 32260 tablet_replica.cc:330] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03: stopping tablet replica
I20260806 09:22:58.671252 32260 raft_consensus.cc:2229] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:58.672005 32260 raft_consensus.cc:2258] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:58.672789 32264 tablet_replica.cc:330] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8: stopping tablet replica
I20260806 09:22:58.673317 32264 raft_consensus.cc:2229] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:22:58.673892 32264 raft_consensus.cc:2258] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:22:58.678344 32261 ts_tablet_manager.cc:1918] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:58.678586 32261 log.cc:1196] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/84645160c0ff4e0894160e303e162bb7
I20260806 09:22:58.679314 32261 ts_tablet_manager.cc:1939] T 84645160c0ff4e0894160e303e162bb7 P fd9824357c5a44bfb6681efdea723cc8: Deleting consensus metadata
Aug 06 09:22:58 dist-test-slave-h3ft krb5kdc[31472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:22:58.682042 31518 catalog_manager.cc:4841] TS fd9824357c5a44bfb6681efdea723cc8 (127.16.86.131:40875): tablet 84645160c0ff4e0894160e303e162bb7 (table test-table [id=27f3095b5e4b4e8f81bdffece5efdc55]) successfully deleted
I20260806 09:22:58.685051 32260 ts_tablet_manager.cc:1905] T 84645160c0ff4e0894160e303e162bb7 P 61c5e3137c3947919e5cfc545e5ada03: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:58.686553 32264 ts_tablet_manager.cc:1905] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:22:58.689317 32216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.657798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35322 (local address 127.16.86.129:39223)
0806 09:22:58.658114 (+ 316us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:58.658124 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:58.669676 (+ 11552us) server_negotiation.cc:407] Connection header received
0806 09:22:58.670307 (+ 631us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:58.670318 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:58.670489 (+ 171us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:58.670595 (+ 106us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:58.672852 (+ 2257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.673789 (+ 937us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.675517 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.675918 (+ 401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.676076 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.680659 (+ 4583us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:58.680677 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:58.680695 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:58.680785 (+ 90us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:58.684564 (+ 3779us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:58.685565 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:58.685577 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:58.685584 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:58.685789 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:58.686304 (+ 515us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:58.686309 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:58.686312 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:58.686876 (+ 564us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:58.687294 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:58.688588 (+ 1294us) server_negotiation.cc:299] Negotiation successful
0806 09:22:58.688871 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20260806 09:22:58.690382 31806 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:58.696003 32264 ts_tablet_manager.cc:1918] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:22:58.696204 32264 log.cc:1196] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/84645160c0ff4e0894160e303e162bb7
I20260806 09:22:58.696686 32264 ts_tablet_manager.cc:1939] T 84645160c0ff4e0894160e303e162bb7 P db7c24c76851428a9fa52e91d27560b8: Deleting consensus metadata
I20260806 09:22:58.698599 31516 catalog_manager.cc:4841] TS db7c24c76851428a9fa52e91d27560b8 (127.16.86.130:43805): tablet 84645160c0ff4e0894160e303e162bb7 (table test-table [id=27f3095b5e4b4e8f81bdffece5efdc55]) successfully deleted
I20260806 09:22:58.856594 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 31628
W20260806 09:22:58.885387 31516 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.86.129:39223 (error 108)
I20260806 09:22:58.885607 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 31819
I20260806 09:22:58.888069 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.887409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39223 (local address 127.0.0.1:35328)
0806 09:22:58.887663 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:58.887828 (+ 165us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260806 09:22:58.888449 31516 catalog_manager.cc:4584] TS 61c5e3137c3947919e5cfc545e5ada03 (127.16.86.129:39223): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 84645160c0ff4e0894160e303e162bb7: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39223: connect: Connection refused (error 111)
Aug 06 09:22:58 dist-test-slave-h3ft krb5kdc[31472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:22:58.903357 32221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.886846 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39070 (local address 127.16.86.130:43805)
0806 09:22:58.887117 (+ 271us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:58.887124 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:58.887146 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:22:58.887406 (+ 260us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:58.887412 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:58.887509 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:58.887662 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:58.889204 (+ 1542us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.890041 (+ 837us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.891522 (+ 1481us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:58.891912 (+ 390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:58.892058 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:58.896359 (+ 4301us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:58.896374 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:58.896391 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:58.896426 (+ 35us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:58.900131 (+ 3705us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:58.900855 (+ 724us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:58.900863 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:58.900866 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:58.900986 (+ 120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:58.901409 (+ 423us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:58.901413 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:58.901415 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:58.901832 (+ 417us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:58.902146 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:58.902487 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 09:22:58.902736 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:22:58.904074 31989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:58.942653 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.941859 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39223 (local address 127.0.0.1:35332)
0806 09:22:58.942231 (+ 372us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:58.942400 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":152}
I20260806 09:22:58.944262 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:58.943688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39223 (local address 127.0.0.1:35340)
0806 09:22:58.943906 (+ 218us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:58.944036 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20260806 09:22:58.999771 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 31819
I20260806 09:22:59.010346 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:59.009268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39223 (local address 127.0.0.1:35354)
0806 09:22:59.009643 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:59.009830 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20260806 09:22:59.012470 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:59.011847 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39223 (local address 127.0.0.1:35364)
0806 09:22:59.012173 (+ 326us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:59.012290 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":190}
I20260806 09:22:59.029623 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 32007
Aug 06 09:22:59 dist-test-slave-h3ft krb5kdc[31472](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008175, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:22:59.050246 32218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:59.030566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45346 (local address 127.16.86.131:40875)
0806 09:22:59.030790 (+ 224us) server_negotiation.cc:206] Beginning negotiation
0806 09:22:59.030796 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:22:59.030944 (+ 148us) server_negotiation.cc:407] Connection header received
0806 09:22:59.031240 (+ 296us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:22:59.031247 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:22:59.031361 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:22:59.031491 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:22:59.032999 (+ 1508us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:59.033815 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:59.035280 (+ 1465us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:22:59.035753 (+ 473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:22:59.035902 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:22:59.040595 (+ 4693us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:22:59.040622 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:22:59.040652 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:22:59.040736 (+ 84us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:22:59.046098 (+ 5362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:59.047270 (+ 1172us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:59.047282 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:59.047286 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:59.047538 (+ 252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:22:59.047992 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:22:59.048002 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:22:59.048006 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:22:59.048682 (+ 676us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:22:59.049176 (+ 494us) server_negotiation.cc:1035] Waiting for connection context
0806 09:22:59.049393 (+ 217us) server_negotiation.cc:299] Negotiation successful
0806 09:22:59.049715 (+ 322us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:22:59.051323 32193 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:59.093787 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:59.093086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39223 (local address 127.0.0.1:35370)
0806 09:22:59.093407 (+ 321us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:59.093538 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":137}
I20260806 09:22:59.095214 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:59.094677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39223 (local address 127.0.0.1:35380)
0806 09:22:59.094929 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:59.095072 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260806 09:22:59.181952 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 32007
I20260806 09:22:59.209848 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 31488
I20260806 09:22:59.211421 31583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:22:59.327848 32217 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:22:59.327290 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39223 (local address 127.0.0.1:35396)
0806 09:22:59.327540 (+ 250us) negotiation.cc:107] Waiting for socket to connect
0806 09:22:59.327643 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39223: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 09:22:59.328145 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 31488
2026-08-06T09:22:59Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5334 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:22:59 dist-test-slave-h3ft krb5kdc[32278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:22:59 dist-test-slave-h3ft krb5kdc[32278](info): set up 2 sockets
Aug 06 09:22:59 dist-test-slave-h3ft krb5kdc[32278](info): commencing operation
krb5kdc: starting...
W20260806 09:23:00.705927 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.321s user 0.001s sys 0.051s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:23:00 dist-test-slave-h3ft krb5kdc[32278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008180, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:23:00Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:23:00Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.86.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.86.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/rangeradmin_127.16.86.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/rangeradmin_127.16.86.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.86.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.86.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/rangerlookup_127.16.86.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/rangerlookup_127.16.86.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.86.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.86.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/HTTP_127.16.86.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/HTTP_127.16.86.148@KRBTEST.COM.keytab.
I20260806 09:23:01.247530 16730 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKAHevx/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/postgres -l logfile start
2026-08-06 09:23:04.567 UTC [32328] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 09:23:04.567 UTC [32328] LOG: listening on IPv4 address "127.16.86.148", port 39885
2026-08-06 09:23:04.573 UTC [32328] LOG: listening on Unix socket "/tmp/.s.PGSQL.39885"
2026-08-06 09:23:04.600 UTC [32333] LOG: database system was shut down at 2026-08-06 09:23:02 UTC
2026-08-06 09:23:04.605 UTC [32328] LOG: database system is ready to accept connections
I20260806 09:23:05.815335 16730 mini_postgres.cc:96] Postgres bound to 39885
2026-08-06 09:23:05.848 UTC [32345] FATAL: database "slave" does not exist
127.16.86.148:39885 - accepting connections
I20260806 09:23:05.849722 16730 mini_ranger.cc:161] Starting Ranger...
I20260806 09:23:05.894626 16730 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 09:23:06.560775 16730 mini_ranger.cc:87] Created ranger Postgres database
I20260806 09:23:06.560946 16730 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-admin
2026-08-06 09:23:06,929 [I] DB FLAVOR :POSTGRES
2026-08-06 09:23:06,930 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 09:23:06,930 [I] Checking connection..
2026-08-06 09:23:06,930 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 09:23:07,209 [I] Checking connection passed.
2026-08-06 09:23:07,210 [I] --------- Verifying version history table ---------
2026-08-06 09:23:07,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:23:07,493 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 09:23:07,493 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:23:07,784 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 09:23:07,785 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 09:23:07,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 09:23:08.084 UTC [32441] WARNING: there is no transaction in progress
2026-08-06 09:23:08,094 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 09:23:08,095 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:23:08,391 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 09:23:08,392 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 09:23:08,392 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 09:23:08,663 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 09:23:08,940 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-h3ft.c.gcp-upstream.internal','N') ;"
2026-08-06 09:23:09,207 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 09:23:09,510 [I] Table x_portal_user does not exist in database ranger
2026-08-06 09:23:09,510 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 09:23:09,797 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 09:23:09,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 09:23:10,061 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 09:23:10,335 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 09:23:10,336 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 09:23:11.464 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.485 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.507 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.524 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.545 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.566 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.633 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.641 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.661 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.674 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.688 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.707 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.720 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.728 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.738 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:11.747 UTC [32610] WARNING: there is no transaction in progress
2026-08-06 09:23:12,579 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 09:23:12,580 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 09:23:12,846 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 09:23:12,847 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 09:23:13,122 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 09:23:13,123 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 09:23:13,427 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 09:23:13,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 09:23:13,691 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 09:23:13,961 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-h3ft.c.gcp-upstream.internal';"
2026-08-06 09:23:14,218 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 09:23:14,218 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 09:23:14,521 [I] DB_PATCHES have already been applied
I20260806 09:23:14.530015 16730 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKAHevx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 09:23:14.530148 16730 mini_ranger.cc:193] Using host: 127.16.86.148
I20260806 09:23:14.559077 16730 mini_ranger.cc:227] Ranger admin URL: http://127.16.86.148:53291
Aug 06, 2026 9:23:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 9:23:14 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 9:23:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 9:23:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 9:23:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 9:23:15 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 9:23:15 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 09:23:16.110008 16730 mini_ranger.cc:160] Time spent starting Ranger: real 10.260s user 0.003s sys 0.089s
Aug 06, 2026 9:23:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 9:23:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 9:23:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 9:23:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 9:23:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 9:23:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 09:23:39.217649 16730 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.86.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.86.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/rangerkms_127.16.86.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/rangerkms_127.16.86.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.86.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.86.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.86.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/HTTP_127.16.86.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.86.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/HTTP_127.16.86.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 09:23:39 dist-test-slave-h3ft krb5kdc[32278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008219, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 09:23:39.483390 16730 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 09:23:39.518749 16730 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 09:23:40.662312 16730 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 09:23:40.662547 16730 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-kms
I20260806 09:23:40.662714 16730 mini_ranger_kms.cc:227] Using postgres at 127.16.86.148:39885
2026-08-06 09:23:41,564 [I] DB FLAVOR :POSTGRES
2026-08-06 09:23:41,565 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 09:23:41,565 [I] Checking connection
2026-08-06 09:23:41,565 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 09:23:41,854 [I] connection success
2026-08-06 09:23:41,854 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 09:23:41,854 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 09:23:41,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 09:23:42,135 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 09:23:42,135 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 09:23:42,135 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 09:23:42,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:39885/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 09:23:42,448 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 09:23:43.062449 16730 mini_ranger_kms.cc:310] Created kms service
I20260806 09:23:43.302619 16730 mini_ranger_kms.cc:326] Created kudu user
I20260806 09:23:43.360587 16730 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 09:23:43 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008219, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:23:43.868314 16730 mini_ranger_kms.cc:384] Added ranger policy
I20260806 09:23:43.868734 16730 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 09:23:43.868790 16730 mini_ranger_kms.cc:240] Using host: 127.16.86.148
I20260806 09:23:43.905993 16730 mini_ranger_kms.cc:276] Ranger KMS PID: 413
I20260806 09:23:43.906301 16730 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.86.148:58047
Aug 06, 2026 9:23:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 9:23:45 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 9:23:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 9:23:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 9:23:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 9:23:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 09:23:46 dist-test-slave-h3ft krb5kdc[32278](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786008226, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.86.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:23:46.787145 16730 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 7.304s user 0.189s sys 0.508s
I20260806 09:23:46.787609 16730 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 09:23:46.787916 16730 mini_ranger_kms.cc:401] 127.16.86.148:58047/kms/v1/keys
Aug 06 09:23:47 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008226, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.86.148@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
Aug 06 09:23:47 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008226, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.86.148@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
Aug 06 09:23:50 dist-test-slave-h3ft krb5kdc[32278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:23:50.228021 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:42529
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:42529
--ranger_config_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.86.148:58047/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:23:50.627755 479 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:23:50.628086 479 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:23:50.628255 479 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:23:50.635667 479 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:23:50.635789 479 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:23:50.635874 479 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:23:50.635965 479 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:23:50.636039 479 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:23:50.646703 479 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.86.148:58047/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:42529
--ranger_config_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:42529
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:23:50.648200 479 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:23:50.649689 479 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:23:50.665539 489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:23:50.665597 491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:23:50.665565 493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:23:50.667239 479 server_base.cc:1034] running on GCE node
I20260806 09:23:50.668475 479 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:23:50.670495 479 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:23:50.671768 479 hybrid_clock.cc:648] HybridClock initialized: now 1786008230671715 us; error 40 us; skew 500 ppm
I20260806 09:23:50.672235 479 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:23:50 dist-test-slave-h3ft krb5kdc[32278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008230, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:23:50.680001 479 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:23:50.682052 479 webserver.cc:460] Webserver started at http://127.16.86.190:35871/ using document root <none> and password file <none>
I20260806 09:23:50.682928 479 fs_manager.cc:359] Metadata directory not provided
I20260806 09:23:50.683120 479 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:23:50.683542 479 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:23:50 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008230, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:23:50.768266 479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "bf4d762726e74497842fa0388d5eea6a"
format_stamp: "Formatted at 2026-08-06 09:23:50 on dist-test-slave-h3ft"
server_key: "dc1a5a94e9d32a8418513fdfdde029fe"
server_key_iv: "10bd58aa86117a119e8651b93138d720"
server_key_version: "kuduclusterkey@0"
I20260806 09:23:50.769321 479 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "bf4d762726e74497842fa0388d5eea6a"
format_stamp: "Formatted at 2026-08-06 09:23:50 on dist-test-slave-h3ft"
server_key: "dc1a5a94e9d32a8418513fdfdde029fe"
server_key_iv: "10bd58aa86117a119e8651b93138d720"
server_key_version: "kuduclusterkey@0"
I20260806 09:23:50.776156 479 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:23:50.801203 501 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:23:50.803658 479 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:23:50.803961 479 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "bf4d762726e74497842fa0388d5eea6a"
format_stamp: "Formatted at 2026-08-06 09:23:50 on dist-test-slave-h3ft"
server_key: "dc1a5a94e9d32a8418513fdfdde029fe"
server_key_iv: "10bd58aa86117a119e8651b93138d720"
server_key_version: "kuduclusterkey@0"
I20260806 09:23:50.804291 479 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:23:50.836922 479 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:23:50.843194 479 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:23:50.843602 479 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:23:50.871857 582 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:42529 every 8 connection(s)
I20260806 09:23:50.871879 479 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:42529
I20260806 09:23:50.874310 479 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:23:50.879091 583 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:23:50.883198 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 479
I20260806 09:23:50.883437 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:23:50.884356 16730 external_mini_cluster.cc:1428] Setting key f63070bec3f900ae327b15f5f7ca03d4
I20260806 09:23:50.895078 583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a: Bootstrap starting.
I20260806 09:23:50.900496 583 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a: Neither blocks nor log segments found. Creating new log.
I20260806 09:23:50.901830 583 log.cc:824] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:23:50.906332 583 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a: No bootstrap required, opened a new log
Aug 06 09:23:50 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:23:50.917822 583 raft_consensus.cc:348] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4d762726e74497842fa0388d5eea6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42529 } }
I20260806 09:23:50.918445 583 raft_consensus.cc:374] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:23:50.918582 583 raft_consensus.cc:729] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf4d762726e74497842fa0388d5eea6a, State: Initialized, Role: FOLLOWER
I20260806 09:23:50.919622 583 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4d762726e74497842fa0388d5eea6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42529 } }
I20260806 09:23:50.919965 583 raft_consensus.cc:388] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:23:50.920147 583 raft_consensus.cc:482] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:23:50.920377 583 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:23:50.924260 583 raft_consensus.cc:504] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4d762726e74497842fa0388d5eea6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42529 } }
I20260806 09:23:50.924993 583 leader_election.cc:302] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf4d762726e74497842fa0388d5eea6a; no voters:
I20260806 09:23:50.928300 587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:50.886934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42806 (local address 127.16.86.190:42529)
0806 09:23:50.900182 (+ 13248us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:50.900218 (+ 36us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:50.901273 (+ 1055us) server_negotiation.cc:407] Connection header received
0806 09:23:50.904688 (+ 3415us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:50.904723 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:50.905287 (+ 564us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:50.905841 (+ 554us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:23:50.907823 (+ 1982us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:50.909616 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:50.911681 (+ 2065us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:50.912814 (+ 1133us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:50.913111 (+ 297us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:50.918149 (+ 5038us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:23:50.918161 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:23:50.918186 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:23:50.918258 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:23:50.922597 (+ 4339us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:50.923586 (+ 989us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:50.923598 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:50.923617 (+ 19us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:50.923834 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:50.924310 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:50.924316 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:50.924321 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:50.925291 (+ 970us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:23:50.925744 (+ 453us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:50.926269 (+ 525us) server_negotiation.cc:299] Negotiation successful
0806 09:23:50.926877 (+ 608us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12713,"thread_start_us":12253,"threads_started":1}
I20260806 09:23:50.930339 583 leader_election.cc:288] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:23:50.930517 589 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:23:50.931586 589 raft_consensus.cc:686] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [term 1 LEADER]: Becoming Leader. State: Replica: bf4d762726e74497842fa0388d5eea6a, State: Running, Role: LEADER
I20260806 09:23:50.932416 589 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4d762726e74497842fa0388d5eea6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42529 } }
I20260806 09:23:50.935491 583 sys_catalog.cc:564] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:23:50.936376 591 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf4d762726e74497842fa0388d5eea6a. Latest consensus state: current_term: 1 leader_uuid: "bf4d762726e74497842fa0388d5eea6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4d762726e74497842fa0388d5eea6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42529 } } }
I20260806 09:23:50.936852 591 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [sys.catalog]: This master's current role is: LEADER
I20260806 09:23:50.941244 590 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf4d762726e74497842fa0388d5eea6a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf4d762726e74497842fa0388d5eea6a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42529 } } }
I20260806 09:23:50.941828 590 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a [sys.catalog]: This master's current role is: LEADER
I20260806 09:23:50.943109 593 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:23:50.949563 593 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:23:50.971699 593 catalog_manager.cc:1347] Generated new cluster ID: fd63bbcaad384f0fa3d2b13b43c2084f
I20260806 09:23:50.971853 593 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:23:50.996915 593 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:23:50.998965 593 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:23:51.029910 593 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a: Generated new TSK 0
I20260806 09:23:51.031100 593 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:23:51.034627 583 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
W20260806 09:23:51.938863 16730 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.86.129: real 0.104s user 0.004s sys 0.026s
I20260806 09:23:51.940474 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.86.148:58047/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42529
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:23:52 dist-test-slave-h3ft krb5kdc[32278](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786008232, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 09:23:52.460180 649 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:23:52.460633 649 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:23:52.460806 649 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:23:52.469717 649 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:23:52.469946 649 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:23:52.470176 649 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:23:52.485996 649 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.86.148:58047/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:23:52.488482 649 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:23:52.490823 649 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:23:52.514137 662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:23:52.515504 664 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:23:52.524406 666 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:23:52.531278 649 server_base.cc:1034] running on GCE node
I20260806 09:23:52.532213 649 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:23:52.533536 649 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:23:52.534979 649 hybrid_clock.cc:648] HybridClock initialized: now 1786008232534884 us; error 81 us; skew 500 ppm
I20260806 09:23:52.535450 649 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:23:52 dist-test-slave-h3ft krb5kdc[32278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008232, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:23:52.539867 649 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:23:52.549147 649 webserver.cc:460] Webserver started at http://127.16.86.129:37725/ using document root <none> and password file <none>
I20260806 09:23:52.550139 649 fs_manager.cc:359] Metadata directory not provided
I20260806 09:23:52.550323 649 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:23:52.550695 649 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:23:52 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008232, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:23:52.579902 649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "cf102ef35bfa4832a056354b052becfa"
format_stamp: "Formatted at 2026-08-06 09:23:52 on dist-test-slave-h3ft"
server_key: "5a709a36f1d11e505cf2cabf2b6bcc60"
server_key_iv: "083565c617516b104862b882b3c39e6a"
server_key_version: "kuduclusterkey@0"
I20260806 09:23:52.581310 649 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "cf102ef35bfa4832a056354b052becfa"
format_stamp: "Formatted at 2026-08-06 09:23:52 on dist-test-slave-h3ft"
server_key: "5a709a36f1d11e505cf2cabf2b6bcc60"
server_key_iv: "083565c617516b104862b882b3c39e6a"
server_key_version: "kuduclusterkey@0"
I20260806 09:23:52.606078 649 fs_manager.cc:691] Time spent creating directory manager: real 0.024s user 0.013s sys 0.010s
I20260806 09:23:52.634480 674 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:23:52.636780 649 fs_manager.cc:725] Time spent opening block manager: real 0.011s user 0.004s sys 0.007s
I20260806 09:23:52.637087 649 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "cf102ef35bfa4832a056354b052becfa"
format_stamp: "Formatted at 2026-08-06 09:23:52 on dist-test-slave-h3ft"
server_key: "5a709a36f1d11e505cf2cabf2b6bcc60"
server_key_iv: "083565c617516b104862b882b3c39e6a"
server_key_version: "kuduclusterkey@0"
I20260806 09:23:52.637573 649 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:23:52.686033 649 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:23:52.692032 649 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:23:52.692487 649 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:23:52.700078 649 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:23:52.708916 649 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:23:52.709102 649 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:23:52.709302 649 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:23:52.709412 649 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:23:52 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008232, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
Aug 06 09:23:53 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008232, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:23:53.502082 851 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:38633 every 8 connection(s)
I20260806 09:23:53.502535 649 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:38633
I20260806 09:23:53.504626 649 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:23:53.515046 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 649
I20260806 09:23:53.515331 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:23:53.516271 16730 external_mini_cluster.cc:1428] Setting key 705ab01cdbfb347a76d8e0950141e64a
Aug 06 09:23:53 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008232, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
I20260806 09:23:53.585716 855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:53.519036 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:36829 (local address 127.16.86.190:42529)
0806 09:23:53.537221 (+ 18185us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:53.537229 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:53.537263 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:23:53.551829 (+ 14566us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:53.551847 (+ 18us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:53.552002 (+ 155us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:53.552148 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:23:53.561586 (+ 9438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:53.562917 (+ 1331us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:53.566177 (+ 3260us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:53.566993 (+ 816us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:53.567166 (+ 173us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:53.574284 (+ 7118us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:23:53.574297 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:23:53.574306 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:23:53.574370 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:23:53.578987 (+ 4617us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:53.581699 (+ 2712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:53.581708 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:53.581712 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:53.581980 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:53.582791 (+ 811us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:53.582797 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:53.582801 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:53.583386 (+ 585us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:23:53.583545 (+ 159us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:53.584889 (+ 1344us) server_negotiation.cc:299] Negotiation successful
0806 09:23:53.585284 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17996,"thread_start_us":23960,"threads_started":1}
I20260806 09:23:53.588631 854 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:53.518251 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:42529 (local address 127.16.86.129:36829)
0806 09:23:53.535708 (+ 17457us) negotiation.cc:107] Waiting for socket to connect
0806 09:23:53.535818 (+ 110us) client_negotiation.cc:174] Beginning negotiation
0806 09:23:53.551217 (+ 15399us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:23:53.559603 (+ 8386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:23:53.559620 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:23:53.560347 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:23:53.561250 (+ 903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:53.561274 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:53.563475 (+ 2201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:53.563484 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:53.565945 (+ 2461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:53.565988 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:53.568635 (+ 2647us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:53.568648 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:53.569148 (+ 500us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:53.570164 (+ 1016us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:23:53.570206 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:23:53.573855 (+ 3649us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:23:53.579340 (+ 5485us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:23:53.579356 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:23:53.579380 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:23:53.580305 (+ 925us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:23:53.582202 (+ 1897us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:23:53.582208 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:23:53.582212 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:23:53.582545 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:23:53.583666 (+ 1121us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:23:53.583676 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:23:53.584240 (+ 564us) client_negotiation.cc:769] Sending connection context
0806 09:23:53.586374 (+ 2134us) client_negotiation.cc:240] Negotiation successful
0806 09:23:53.587087 (+ 713us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16918,"thread_start_us":16565,"threads_started":1}
I20260806 09:23:53.590994 852 heartbeater.cc:347] Connected to a master server at 127.16.86.190:42529
I20260806 09:23:53.591408 852 heartbeater.cc:464] Registering TS with master...
I20260806 09:23:53.592271 852 heartbeater.cc:511] Master 127.16.86.190:42529 requested a full tablet report, sending...
I20260806 09:23:53.595329 524 ts_manager.cc:194] Registered new tserver with Master: cf102ef35bfa4832a056354b052becfa (127.16.86.129:38633)
I20260806 09:23:53.597555 524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:36829
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
W20260806 09:23:53.638563 16730 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.86.130: real 0.119s user 0.000s sys 0.033s
I20260806 09:23:53.639843 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.86.148:58047/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42529
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:23:54.183746 860 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:23:54.184206 860 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:23:54.184401 860 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:23:54.194334 860 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:23:54.194567 860 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:23:54.194722 860 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:23:54.208298 860 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.86.148:58047/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:23:54.210323 860 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:23:54.212414 860 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:23:54.230439 875 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:23:54.230914 878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:23:54.231104 874 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:23:54.231933 860 server_base.cc:1034] running on GCE node
I20260806 09:23:54.232798 860 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:23:54.234270 860 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:23:54.235562 860 hybrid_clock.cc:648] HybridClock initialized: now 1786008234235457 us; error 47 us; skew 500 ppm
I20260806 09:23:54.236166 860 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:23:54 dist-test-slave-h3ft krb5kdc[32278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:23:54.240577 860 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:23:54.243425 860 webserver.cc:460] Webserver started at http://127.16.86.130:35763/ using document root <none> and password file <none>
I20260806 09:23:54.244539 860 fs_manager.cc:359] Metadata directory not provided
I20260806 09:23:54.244778 860 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:23:54.245298 860 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:23:54 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:23:54.263161 860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "d6a1acc0aa5b4f07bb2335217635a49f"
format_stamp: "Formatted at 2026-08-06 09:23:54 on dist-test-slave-h3ft"
server_key: "3c44d9adc0e86672620a10c856f44cee"
server_key_iv: "969ca1f34d0a41478231700d83719bc0"
server_key_version: "kuduclusterkey@0"
I20260806 09:23:54.264454 860 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "d6a1acc0aa5b4f07bb2335217635a49f"
format_stamp: "Formatted at 2026-08-06 09:23:54 on dist-test-slave-h3ft"
server_key: "3c44d9adc0e86672620a10c856f44cee"
server_key_iv: "969ca1f34d0a41478231700d83719bc0"
server_key_version: "kuduclusterkey@0"
I20260806 09:23:54.272415 860 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260806 09:23:54.289331 886 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:23:54.291493 860 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:23:54.291749 860 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "d6a1acc0aa5b4f07bb2335217635a49f"
format_stamp: "Formatted at 2026-08-06 09:23:54 on dist-test-slave-h3ft"
server_key: "3c44d9adc0e86672620a10c856f44cee"
server_key_iv: "969ca1f34d0a41478231700d83719bc0"
server_key_version: "kuduclusterkey@0"
I20260806 09:23:54.292038 860 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:23:54.324720 860 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:23:54.329528 860 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:23:54.329896 860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:23:54.331300 860 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:23:54.333525 860 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:23:54.333662 860 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:23:54.333864 860 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:23:54.333962 860 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:23:54.403174 860 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:36139
I20260806 09:23:54.403190 1061 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:36139 every 8 connection(s)
I20260806 09:23:54.404806 860 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:23:54.410315 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 860
I20260806 09:23:54.410473 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:23:54.411144 16730 external_mini_cluster.cc:1428] Setting key 166ef387eac24c5848203ae27cde66c4
Aug 06 09:23:54 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008234, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:23:54.436993 1064 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:54.408851 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:60871 (local address 127.16.86.190:42529)
0806 09:23:54.409803 (+ 952us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:54.409812 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:54.412480 (+ 2668us) server_negotiation.cc:407] Connection header received
0806 09:23:54.415392 (+ 2912us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:54.415402 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:54.415513 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:54.415674 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:23:54.418060 (+ 2386us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:54.419770 (+ 1710us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:54.422054 (+ 2284us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:54.422952 (+ 898us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:54.423145 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:54.428835 (+ 5690us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:23:54.428841 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:23:54.428845 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:23:54.428911 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:23:54.432403 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:54.433589 (+ 1186us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:54.433593 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:54.433637 (+ 44us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:54.433790 (+ 153us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:54.434440 (+ 650us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:54.434443 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:54.434445 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:54.434809 (+ 364us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:23:54.434947 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:54.436289 (+ 1342us) server_negotiation.cc:299] Negotiation successful
0806 09:23:54.436575 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":694,"thread_start_us":494,"threads_started":1}
I20260806 09:23:54.439106 1067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:54.410191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:42529 (local address 127.16.86.130:60871)
0806 09:23:54.412239 (+ 2048us) negotiation.cc:107] Waiting for socket to connect
0806 09:23:54.412299 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:23:54.414963 (+ 2664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:23:54.416051 (+ 1088us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:23:54.416062 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:23:54.416840 (+ 778us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:23:54.417822 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:54.417853 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:54.419981 (+ 2128us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:54.419986 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:54.421875 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:54.421891 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:54.423470 (+ 1579us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:54.423481 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:54.423885 (+ 404us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:54.424964 (+ 1079us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:23:54.424991 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:23:54.428520 (+ 3529us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:23:54.432588 (+ 4068us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:23:54.432602 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:23:54.432626 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:23:54.433440 (+ 814us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:23:54.433932 (+ 492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:23:54.433940 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:23:54.433946 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:23:54.434303 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:23:54.434985 (+ 682us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:23:54.434998 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:23:54.435658 (+ 660us) client_negotiation.cc:769] Sending connection context
0806 09:23:54.436055 (+ 397us) client_negotiation.cc:240] Negotiation successful
0806 09:23:54.436863 (+ 808us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1627,"thread_start_us":1179,"threads_started":1}
I20260806 09:23:54.441128 1062 heartbeater.cc:347] Connected to a master server at 127.16.86.190:42529
I20260806 09:23:54.441722 1062 heartbeater.cc:464] Registering TS with master...
I20260806 09:23:54.442919 1062 heartbeater.cc:511] Master 127.16.86.190:42529 requested a full tablet report, sending...
I20260806 09:23:54.445132 524 ts_manager.cc:194] Registered new tserver with Master: d6a1acc0aa5b4f07bb2335217635a49f (127.16.86.130:36139)
I20260806 09:23:54.446183 524 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:60871
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:23:54.499663 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.86.148:58047/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42529
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:36713
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:23:54.602449 852 heartbeater.cc:503] Master 127.16.86.190:42529 was elected leader, sending a full tablet report...
W20260806 09:23:55.123663 1072 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:23:55.124220 1072 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:23:55.124495 1072 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:23:55.135157 1072 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:23:55.135447 1072 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:23:55.135671 1072 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:23:55.149919 1072 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:36713
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.86.148:58047/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42529
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:23:55.151958 1072 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:23:55.154109 1072 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:23:55.176762 1078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:23:55.177947 1083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:23:55.178045 1079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:23:55.180305 1072 server_base.cc:1034] running on GCE node
I20260806 09:23:55.181303 1072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:23:55.182830 1072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:23:55.184154 1072 hybrid_clock.cc:648] HybridClock initialized: now 1786008235184056 us; error 90 us; skew 500 ppm
I20260806 09:23:55.184643 1072 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:23:55 dist-test-slave-h3ft krb5kdc[32278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008235, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:23:55.189826 1072 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:23:55.192337 1072 webserver.cc:460] Webserver started at http://127.16.86.131:39571/ using document root <none> and password file <none>
I20260806 09:23:55.193410 1072 fs_manager.cc:359] Metadata directory not provided
I20260806 09:23:55.193666 1072 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:23:55.194203 1072 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:23:55 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008235, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:23:55.215668 1072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "b618609594424de5a460e036b881fadd"
format_stamp: "Formatted at 2026-08-06 09:23:55 on dist-test-slave-h3ft"
server_key: "b0cc7290e84091a96e07dc73cecbe2ef"
server_key_iv: "ae497b90b6b641c497885eb3d5fdac79"
server_key_version: "kuduclusterkey@0"
I20260806 09:23:55.216765 1072 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "b618609594424de5a460e036b881fadd"
format_stamp: "Formatted at 2026-08-06 09:23:55 on dist-test-slave-h3ft"
server_key: "b0cc7290e84091a96e07dc73cecbe2ef"
server_key_iv: "ae497b90b6b641c497885eb3d5fdac79"
server_key_version: "kuduclusterkey@0"
I20260806 09:23:55.225167 1072 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.008s sys 0.000s
I20260806 09:23:55.246770 1096 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:23:55.249348 1072 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.002s
I20260806 09:23:55.249666 1072 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "b618609594424de5a460e036b881fadd"
format_stamp: "Formatted at 2026-08-06 09:23:55 on dist-test-slave-h3ft"
server_key: "b0cc7290e84091a96e07dc73cecbe2ef"
server_key_iv: "ae497b90b6b641c497885eb3d5fdac79"
server_key_version: "kuduclusterkey@0"
I20260806 09:23:55.250105 1072 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:23:55.280085 1072 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:23:55.285789 1072 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:23:55.286306 1072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:23:55.287941 1072 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:23:55.290030 1072 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:23:55.290122 1072 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:23:55.290246 1072 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:23:55.290304 1072 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:23:55.365267 1072 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:34469
I20260806 09:23:55.365315 1275 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:34469 every 8 connection(s)
I20260806 09:23:55.367759 1072 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:23:55.374567 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 1072
I20260806 09:23:55.374840 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:23:55.375484 16730 external_mini_cluster.cc:1428] Setting key 9ae658bac26abb83442df659e4e1c8c5
Aug 06 09:23:55 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008235, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:23:55.400390 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.372439 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:35589 (local address 127.16.86.190:42529)
0806 09:23:55.373563 (+ 1124us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:55.373568 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:55.375376 (+ 1808us) server_negotiation.cc:407] Connection header received
0806 09:23:55.378932 (+ 3556us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:55.378940 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:55.379199 (+ 259us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:55.379376 (+ 177us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:23:55.381701 (+ 2325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.382788 (+ 1087us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.385117 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.385835 (+ 718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.385977 (+ 142us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.390757 (+ 4780us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:23:55.390763 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:23:55.390767 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:23:55.390812 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:23:55.395972 (+ 5160us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:55.396999 (+ 1027us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:55.397004 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:55.397008 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:55.397233 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:55.397919 (+ 686us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:55.397925 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:55.397928 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:55.398449 (+ 521us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:23:55.398592 (+ 143us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:55.399652 (+ 1060us) server_negotiation.cc:299] Negotiation successful
0806 09:23:55.399898 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":994,"thread_start_us":755,"threads_started":1}
I20260806 09:23:55.401476 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.373253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:42529 (local address 127.16.86.131:35589)
0806 09:23:55.375120 (+ 1867us) negotiation.cc:107] Waiting for socket to connect
0806 09:23:55.375186 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 09:23:55.378471 (+ 3285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:23:55.379597 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:23:55.379606 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:23:55.380534 (+ 928us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:23:55.381500 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:55.381520 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.383049 (+ 1529us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.383057 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:55.384917 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:55.384930 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.386155 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.386161 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:55.386420 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.387341 (+ 921us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:23:55.387365 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:23:55.390424 (+ 3059us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:23:55.396169 (+ 5745us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:23:55.396177 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:23:55.396194 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:23:55.396835 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:23:55.397392 (+ 557us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:23:55.397396 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:23:55.397399 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:23:55.397729 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:23:55.398598 (+ 869us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:23:55.398607 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:23:55.399200 (+ 593us) client_negotiation.cc:769] Sending connection context
0806 09:23:55.399496 (+ 296us) client_negotiation.cc:240] Negotiation successful
0806 09:23:55.400002 (+ 506us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1376,"thread_start_us":768,"threads_started":1}
I20260806 09:23:55.404026 1276 heartbeater.cc:347] Connected to a master server at 127.16.86.190:42529
I20260806 09:23:55.404489 1276 heartbeater.cc:464] Registering TS with master...
I20260806 09:23:55.405851 1276 heartbeater.cc:511] Master 127.16.86.190:42529 requested a full tablet report, sending...
I20260806 09:23:55.408774 522 ts_manager.cc:194] Registered new tserver with Master: b618609594424de5a460e036b881fadd (127.16.86.131:34469)
I20260806 09:23:55.411074 522 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:35589
I20260806 09:23:55.421561 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:23:55.448861 1281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.428351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42822 (local address 127.16.86.190:42529)
0806 09:23:55.428590 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:55.428595 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:55.429490 (+ 895us) server_negotiation.cc:407] Connection header received
0806 09:23:55.429701 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:55.429706 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:55.429891 (+ 185us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:55.430031 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:23:55.432429 (+ 2398us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.433598 (+ 1169us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.435693 (+ 2095us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.436477 (+ 784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.436669 (+ 192us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.440454 (+ 3785us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:23:55.440461 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:23:55.440465 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:23:55.440518 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:23:55.445021 (+ 4503us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:55.446019 (+ 998us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:55.446024 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:55.446027 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:55.446252 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:55.446843 (+ 591us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:55.446849 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:55.446853 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:55.447478 (+ 625us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:23:55.447774 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:55.448159 (+ 385us) server_negotiation.cc:299] Negotiation successful
0806 09:23:55.448469 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:23:55.449822 1062 heartbeater.cc:503] Master 127.16.86.190:42529 was elected leader, sending a full tablet report...
I20260806 09:23:55.454974 522 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42822:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:23:55.466367 522 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:23:55.514859 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.487588 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37816 (local address 127.16.86.129:38633)
0806 09:23:55.500960 (+ 13372us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:55.500971 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:55.501002 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:23:55.501122 (+ 120us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:55.501130 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:55.501432 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:55.501980 (+ 548us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:23:55.504783 (+ 2803us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.507260 (+ 2477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.511573 (+ 4313us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.513095 (+ 1522us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.513288 (+ 193us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.513635 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:55.513992 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 09:23:55.514199 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13096,"thread_start_us":13333,"threads_started":1}
I20260806 09:23:55.515120 1297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.486786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:38633 (local address 127.0.0.1:37816)
0806 09:23:55.493849 (+ 7063us) negotiation.cc:107] Waiting for socket to connect
0806 09:23:55.493883 (+ 34us) client_negotiation.cc:174] Beginning negotiation
0806 09:23:55.494047 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:23:55.501801 (+ 7754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:23:55.501811 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:23:55.501894 (+ 83us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:23:55.504380 (+ 2486us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:55.504393 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.507451 (+ 3058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.507457 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:55.511363 (+ 3906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:55.511379 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.513434 (+ 2055us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.513442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:55.513738 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.513783 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 09:23:55.513928 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 09:23:55.514189 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6909,"mutex_wait_us":175,"spinlock_wait_cycles":4992,"thread_start_us":869,"threads_started":1}
I20260806 09:23:55.516022 1299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.486314 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:36139 (local address 127.0.0.1:51394)
0806 09:23:55.488462 (+ 2148us) negotiation.cc:107] Waiting for socket to connect
0806 09:23:55.488523 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 09:23:55.488818 (+ 295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:23:55.501808 (+ 12990us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:23:55.501814 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:23:55.501929 (+ 115us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:23:55.504182 (+ 2253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:55.504203 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.507413 (+ 3210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.507421 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:55.511560 (+ 4139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:55.511569 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.514429 (+ 2860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.514438 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:55.514705 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.514789 (+ 84us) client_negotiation.cc:769] Sending connection context
0806 09:23:55.514921 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 09:23:55.515113 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1747,"mutex_wait_us":267,"spinlock_wait_cycles":1792,"thread_start_us":7206,"threads_started":1}
I20260806 09:23:55.516503 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.486820 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51394 (local address 127.16.86.130:36139)
0806 09:23:55.500973 (+ 14153us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:55.500985 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:55.501023 (+ 38us) server_negotiation.cc:407] Connection header received
0806 09:23:55.501153 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:55.501162 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:55.501523 (+ 361us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:55.502381 (+ 858us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:23:55.504465 (+ 2084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.507220 (+ 2755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.511838 (+ 4618us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.513962 (+ 2124us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.515344 (+ 1382us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.515626 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:55.515785 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 09:23:55.516013 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13892,"thread_start_us":14874,"threads_started":1}
I20260806 09:23:55.517717 1304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.489068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:34469 (local address 127.0.0.1:51766)
0806 09:23:55.499354 (+ 10286us) negotiation.cc:107] Waiting for socket to connect
0806 09:23:55.499414 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:23:55.499665 (+ 251us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:23:55.501801 (+ 2136us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:23:55.501811 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:23:55.501866 (+ 55us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:23:55.503548 (+ 1682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:55.503564 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.506893 (+ 3329us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.506899 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:55.511548 (+ 4649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:55.511566 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.516378 (+ 4812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.516391 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:55.516647 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.516699 (+ 52us) client_negotiation.cc:769] Sending connection context
0806 09:23:55.516777 (+ 78us) client_negotiation.cc:240] Negotiation successful
0806 09:23:55.516982 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10059,"mutex_wait_us":299,"spinlock_wait_cycles":896,"thread_start_us":10779,"threads_started":1}
I20260806 09:23:55.518848 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.489690 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51766 (local address 127.16.86.131:34469)
0806 09:23:55.500215 (+ 10525us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:55.500226 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:55.500271 (+ 45us) server_negotiation.cc:407] Connection header received
0806 09:23:55.500435 (+ 164us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:55.500448 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:55.500881 (+ 433us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:55.501178 (+ 297us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:23:55.504302 (+ 3124us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.506700 (+ 2398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.511828 (+ 5128us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.513963 (+ 2135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.514144 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.514610 (+ 466us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:55.518191 (+ 3581us) server_negotiation.cc:299] Negotiation successful
0806 09:23:55.518477 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10275,"thread_start_us":9935,"threads_started":1}
I20260806 09:23:55.521131 749 tablet_service.cc:1467] Processing CreateTablet for tablet bd8ab27870b84185b6835c8ea1c461fd (DEFAULT_TABLE table=test-table [id=488deff3ec964ad69b8eca0bee4ac123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:23:55.521469 964 tablet_service.cc:1467] Processing CreateTablet for tablet bd8ab27870b84185b6835c8ea1c461fd (DEFAULT_TABLE table=test-table [id=488deff3ec964ad69b8eca0bee4ac123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:23:55.523212 749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8ab27870b84185b6835c8ea1c461fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:23:55.523586 964 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8ab27870b84185b6835c8ea1c461fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:23:55.522908 1175 tablet_service.cc:1467] Processing CreateTablet for tablet bd8ab27870b84185b6835c8ea1c461fd (DEFAULT_TABLE table=test-table [id=488deff3ec964ad69b8eca0bee4ac123]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:23:55.525290 1175 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet bd8ab27870b84185b6835c8ea1c461fd. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:23:55.548413 1308 tablet_bootstrap.cc:492] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd: Bootstrap starting.
I20260806 09:23:55.549033 1309 tablet_bootstrap.cc:492] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f: Bootstrap starting.
I20260806 09:23:55.554188 1308 tablet_bootstrap.cc:654] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd: Neither blocks nor log segments found. Creating new log.
I20260806 09:23:55.557587 1309 tablet_bootstrap.cc:654] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f: Neither blocks nor log segments found. Creating new log.
I20260806 09:23:55.557621 1308 log.cc:824] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd: Log is configured to *not* fsync() on all Append() calls
I20260806 09:23:55.560315 1309 log.cc:824] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:23:55.560715 1307 tablet_bootstrap.cc:492] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa: Bootstrap starting.
I20260806 09:23:55.564755 1309 tablet_bootstrap.cc:492] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f: No bootstrap required, opened a new log
I20260806 09:23:55.565155 1309 ts_tablet_manager.cc:1397] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f: Time spent bootstrapping tablet: real 0.017s user 0.012s sys 0.000s
I20260806 09:23:55.566562 1307 tablet_bootstrap.cc:654] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa: Neither blocks nor log segments found. Creating new log.
I20260806 09:23:55.566689 1308 tablet_bootstrap.cc:492] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd: No bootstrap required, opened a new log
I20260806 09:23:55.567274 1308 ts_tablet_manager.cc:1397] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd: Time spent bootstrapping tablet: real 0.019s user 0.003s sys 0.010s
I20260806 09:23:55.569031 1307 log.cc:824] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa: Log is configured to *not* fsync() on all Append() calls
I20260806 09:23:55.576164 1307 tablet_bootstrap.cc:492] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa: No bootstrap required, opened a new log
I20260806 09:23:55.576812 1307 ts_tablet_manager.cc:1397] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.007s
I20260806 09:23:55.577898 1309 raft_consensus.cc:348] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1acc0aa5b4f07bb2335217635a49f" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 36139 } } peers { permanent_uuid: "cf102ef35bfa4832a056354b052becfa" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 38633 } } peers { permanent_uuid: "b618609594424de5a460e036b881fadd" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34469 } }
I20260806 09:23:55.578810 1309 raft_consensus.cc:374] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:23:55.579031 1309 raft_consensus.cc:729] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6a1acc0aa5b4f07bb2335217635a49f, State: Initialized, Role: FOLLOWER
I20260806 09:23:55.580166 1309 consensus_queue.cc:260] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1acc0aa5b4f07bb2335217635a49f" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 36139 } } peers { permanent_uuid: "cf102ef35bfa4832a056354b052becfa" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 38633 } } peers { permanent_uuid: "b618609594424de5a460e036b881fadd" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34469 } }
I20260806 09:23:55.582443 1308 raft_consensus.cc:348] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1acc0aa5b4f07bb2335217635a49f" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 36139 } } peers { permanent_uuid: "cf102ef35bfa4832a056354b052becfa" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 38633 } } peers { permanent_uuid: "b618609594424de5a460e036b881fadd" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34469 } }
I20260806 09:23:55.583384 1309 ts_tablet_manager.cc:1428] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f: Time spent starting tablet: real 0.018s user 0.012s sys 0.008s
I20260806 09:23:55.583556 1308 raft_consensus.cc:374] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:23:55.583909 1308 raft_consensus.cc:729] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b618609594424de5a460e036b881fadd, State: Initialized, Role: FOLLOWER
I20260806 09:23:55.585273 1308 consensus_queue.cc:260] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1acc0aa5b4f07bb2335217635a49f" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 36139 } } peers { permanent_uuid: "cf102ef35bfa4832a056354b052becfa" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 38633 } } peers { permanent_uuid: "b618609594424de5a460e036b881fadd" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34469 } }
I20260806 09:23:55.593900 1276 heartbeater.cc:503] Master 127.16.86.190:42529 was elected leader, sending a full tablet report...
I20260806 09:23:55.595117 1308 ts_tablet_manager.cc:1428] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd: Time spent starting tablet: real 0.028s user 0.026s sys 0.000s
I20260806 09:23:55.599846 1307 raft_consensus.cc:348] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1acc0aa5b4f07bb2335217635a49f" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 36139 } } peers { permanent_uuid: "cf102ef35bfa4832a056354b052becfa" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 38633 } } peers { permanent_uuid: "b618609594424de5a460e036b881fadd" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34469 } }
I20260806 09:23:55.601440 1307 raft_consensus.cc:374] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:23:55.601613 1307 raft_consensus.cc:729] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cf102ef35bfa4832a056354b052becfa, State: Initialized, Role: FOLLOWER
I20260806 09:23:55.602600 1307 consensus_queue.cc:260] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1acc0aa5b4f07bb2335217635a49f" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 36139 } } peers { permanent_uuid: "cf102ef35bfa4832a056354b052becfa" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 38633 } } peers { permanent_uuid: "b618609594424de5a460e036b881fadd" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34469 } }
I20260806 09:23:55.609512 1307 ts_tablet_manager.cc:1428] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa: Time spent starting tablet: real 0.032s user 0.019s sys 0.004s
W20260806 09:23:55.621333 1277 tablet.cc:2369] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:23:55.658396 1063 tablet.cc:2369] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:23:55.791246 1317 raft_consensus.cc:482] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:23:55.791716 1317 raft_consensus.cc:504] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1acc0aa5b4f07bb2335217635a49f" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 36139 } } peers { permanent_uuid: "cf102ef35bfa4832a056354b052becfa" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 38633 } } peers { permanent_uuid: "b618609594424de5a460e036b881fadd" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34469 } }
I20260806 09:23:55.794131 1317 leader_election.cc:288] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6a1acc0aa5b4f07bb2335217635a49f (127.16.86.130:36139), cf102ef35bfa4832a056354b052becfa (127.16.86.129:38633)
I20260806 09:23:55.803119 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.795094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:48669 (local address 127.16.86.129:38633)
0806 09:23:55.795412 (+ 318us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:55.795417 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:55.795444 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:23:55.795570 (+ 126us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:55.795578 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:55.795735 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:55.795881 (+ 146us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:23:55.796555 (+ 674us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.797944 (+ 1389us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.800627 (+ 2683us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.801950 (+ 1323us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.802088 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.802147 (+ 59us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:55.802648 (+ 501us) server_negotiation.cc:299] Negotiation successful
0806 09:23:55.802777 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":133}
I20260806 09:23:55.803310 1283 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.794630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:38633 (local address 127.16.86.131:48669)
0806 09:23:55.794898 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 09:23:55.794930 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 09:23:55.795148 (+ 218us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:23:55.795971 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:23:55.795982 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:23:55.796066 (+ 84us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:23:55.796363 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:55.796396 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.798139 (+ 1743us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.798146 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:55.800357 (+ 2211us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:55.800374 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.802161 (+ 1787us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.802169 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:55.802406 (+ 237us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.802456 (+ 50us) client_negotiation.cc:769] Sending connection context
0806 09:23:55.802597 (+ 141us) client_negotiation.cc:240] Negotiation successful
0806 09:23:55.802798 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":68,"mutex_wait_us":1}
I20260806 09:23:55.804715 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.794764 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:42251 (local address 127.16.86.130:36139)
0806 09:23:55.794990 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:55.794994 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:55.796289 (+ 1295us) server_negotiation.cc:407] Connection header received
0806 09:23:55.796482 (+ 193us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:55.796492 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:55.796608 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:55.796700 (+ 92us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:23:55.797650 (+ 950us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.799058 (+ 1408us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.801682 (+ 2624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.803292 (+ 1610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.803536 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.803632 (+ 96us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:55.804125 (+ 493us) server_negotiation.cc:299] Negotiation successful
0806 09:23:55.804337 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:23:55.804708 1321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.794647 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:36139 (local address 127.16.86.131:42251)
0806 09:23:55.796131 (+ 1484us) negotiation.cc:107] Waiting for socket to connect
0806 09:23:55.796159 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:23:55.796317 (+ 158us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:23:55.796786 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:23:55.796793 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:23:55.796833 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:23:55.797436 (+ 603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:55.797451 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.799262 (+ 1811us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.799266 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:55.801436 (+ 2170us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:23:55.801456 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.803632 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.803639 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:23:55.803875 (+ 236us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.803916 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:23:55.804036 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 09:23:55.804185 (+ 149us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1215,"mutex_wait_us":553,"thread_start_us":411,"threads_started":1}
I20260806 09:23:55.804999 785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8ab27870b84185b6835c8ea1c461fd" candidate_uuid: "b618609594424de5a460e036b881fadd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf102ef35bfa4832a056354b052becfa" is_pre_election: true
I20260806 09:23:55.805990 785 raft_consensus.cc:2454] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b618609594424de5a460e036b881fadd in term 0.
I20260806 09:23:55.806581 995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8ab27870b84185b6835c8ea1c461fd" candidate_uuid: "b618609594424de5a460e036b881fadd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1acc0aa5b4f07bb2335217635a49f" is_pre_election: true
I20260806 09:23:55.807088 1102 leader_election.cc:302] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: b618609594424de5a460e036b881fadd, cf102ef35bfa4832a056354b052becfa; no voters:
I20260806 09:23:55.807480 995 raft_consensus.cc:2454] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b618609594424de5a460e036b881fadd in term 0.
I20260806 09:23:55.807732 1317 raft_consensus.cc:2781] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:23:55.807956 1317 raft_consensus.cc:482] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:23:55.808172 1317 raft_consensus.cc:3037] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:23:55.810966 1317 raft_consensus.cc:504] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1acc0aa5b4f07bb2335217635a49f" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 36139 } } peers { permanent_uuid: "cf102ef35bfa4832a056354b052becfa" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 38633 } } peers { permanent_uuid: "b618609594424de5a460e036b881fadd" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34469 } }
I20260806 09:23:55.811856 1317 leader_election.cc:288] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [CANDIDATE]: Term 1 election: Requested vote from peers d6a1acc0aa5b4f07bb2335217635a49f (127.16.86.130:36139), cf102ef35bfa4832a056354b052becfa (127.16.86.129:38633)
I20260806 09:23:55.812724 785 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8ab27870b84185b6835c8ea1c461fd" candidate_uuid: "b618609594424de5a460e036b881fadd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cf102ef35bfa4832a056354b052becfa"
I20260806 09:23:55.812775 995 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "bd8ab27870b84185b6835c8ea1c461fd" candidate_uuid: "b618609594424de5a460e036b881fadd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6a1acc0aa5b4f07bb2335217635a49f"
I20260806 09:23:55.813026 785 raft_consensus.cc:3037] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:23:55.813086 995 raft_consensus.cc:3037] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f [term 0 FOLLOWER]: Advancing to term 1
W20260806 09:23:55.814741 853 tablet.cc:2369] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:23:55.817269 785 raft_consensus.cc:2454] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b618609594424de5a460e036b881fadd in term 1.
I20260806 09:23:55.817240 995 raft_consensus.cc:2454] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b618609594424de5a460e036b881fadd in term 1.
I20260806 09:23:55.818071 1099 leader_election.cc:302] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: b618609594424de5a460e036b881fadd, cf102ef35bfa4832a056354b052becfa, d6a1acc0aa5b4f07bb2335217635a49f; no voters:
I20260806 09:23:55.818533 1317 raft_consensus.cc:2781] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:23:55.819142 1317 raft_consensus.cc:686] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 1 LEADER]: Becoming Leader. State: Replica: b618609594424de5a460e036b881fadd, State: Running, Role: LEADER
I20260806 09:23:55.819779 1317 consensus_queue.cc:237] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1acc0aa5b4f07bb2335217635a49f" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 36139 } } peers { permanent_uuid: "cf102ef35bfa4832a056354b052becfa" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 38633 } } peers { permanent_uuid: "b618609594424de5a460e036b881fadd" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34469 } }
I20260806 09:23:55.825784 522 catalog_manager.cc:5495] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd reported cstate change: term changed from 0 to 1, leader changed from <none> to b618609594424de5a460e036b881fadd (127.16.86.131). New cstate: current_term: 1 leader_uuid: "b618609594424de5a460e036b881fadd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6a1acc0aa5b4f07bb2335217635a49f" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 36139 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cf102ef35bfa4832a056354b052becfa" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 38633 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b618609594424de5a460e036b881fadd" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34469 } health_report { overall_health: HEALTHY } } }
I20260806 09:23:55.896462 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.887977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51778 (local address 127.16.86.131:34469)
0806 09:23:55.888307 (+ 330us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:55.888312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:55.888336 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:23:55.888589 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:55.888594 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:55.888712 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:55.888873 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:23:55.889445 (+ 572us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.890529 (+ 1084us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.893227 (+ 2698us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.893741 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.893856 (+ 115us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:55.894744 (+ 888us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:23:55.895606 (+ 862us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:23:55.895745 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:55.895931 (+ 186us) server_negotiation.cc:299] Negotiation successful
0806 09:23:55.896031 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:23:55.908351 785 raft_consensus.cc:1261] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa [term 1 FOLLOWER]: Refusing update from remote peer b618609594424de5a460e036b881fadd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:23:55.908528 995 raft_consensus.cc:1261] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f [term 1 FOLLOWER]: Refusing update from remote peer b618609594424de5a460e036b881fadd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:23:55.909955 1322 consensus_queue.cc:1035] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [LEADER]: Connected to new peer: Peer: permanent_uuid: "cf102ef35bfa4832a056354b052becfa" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 38633 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:23:55.910481 1317 consensus_queue.cc:1035] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6a1acc0aa5b4f07bb2335217635a49f" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 36139 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:23:55.927882 1325 mvcc.cc:205] Tried to move back new op lower bound from 7315489734265106432 to 7315489733922951168. Current Snapshot: MvccSnapshot[applied={T|T < 7315489734265106432}]
I20260806 09:23:55.929437 1328 mvcc.cc:205] Tried to move back new op lower bound from 7315489734265106432 to 7315489733922951168. Current Snapshot: MvccSnapshot[applied={T|T < 7315489734265106432}]
I20260806 09:23:55.932726 1330 mvcc.cc:205] Tried to move back new op lower bound from 7315489734265106432 to 7315489733922951168. Current Snapshot: MvccSnapshot[applied={T|T < 7315489734265106432}]
I20260806 09:23:55.950939 522 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:23:55.951469 522 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42822:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:23:55.956490 522 catalog_manager.cc:5782] T 00000000000000000000000000000000 P bf4d762726e74497842fa0388d5eea6a: Sending DeleteTablet for 3 replicas of tablet bd8ab27870b84185b6835c8ea1c461fd
I20260806 09:23:55.958209 964 tablet_service.cc:1514] Processing DeleteTablet for tablet bd8ab27870b84185b6835c8ea1c461fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:23:55 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:51394
I20260806 09:23:55.958369 749 tablet_service.cc:1514] Processing DeleteTablet for tablet bd8ab27870b84185b6835c8ea1c461fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:23:55 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:37816
I20260806 09:23:55.958663 1175 tablet_service.cc:1514] Processing DeleteTablet for tablet bd8ab27870b84185b6835c8ea1c461fd with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:23:55 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:51766
I20260806 09:23:55.962001 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 649
I20260806 09:23:55.979552 1341 tablet_replica.cc:330] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd: stopping tablet replica
I20260806 09:23:55.980123 1341 raft_consensus.cc:2229] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:23:55.980897 1341 raft_consensus.cc:2258] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:23:55.982295 1340 tablet_replica.cc:330] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa: stopping tablet replica
I20260806 09:23:55.983439 1339 tablet_replica.cc:330] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f: stopping tablet replica
I20260806 09:23:55.983579 1341 ts_tablet_manager.cc:1905] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:23:55.983879 1339 raft_consensus.cc:2229] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:23:55.985522 1339 raft_consensus.cc:2258] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:23:55.982762 1340 raft_consensus.cc:2229] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:23:55.987308 1340 raft_consensus.cc:2258] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:23:55.989645 1340 ts_tablet_manager.cc:1905] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:23:55.990397 1339 ts_tablet_manager.cc:1905] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:23:55.991950 1341 ts_tablet_manager.cc:1918] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:23:55.992141 1341 log.cc:1196] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/bd8ab27870b84185b6835c8ea1c461fd
I20260806 09:23:55.992980 1341 ts_tablet_manager.cc:1939] T bd8ab27870b84185b6835c8ea1c461fd P b618609594424de5a460e036b881fadd: Deleting consensus metadata
I20260806 09:23:55.995658 505 catalog_manager.cc:4841] TS b618609594424de5a460e036b881fadd (127.16.86.131:34469): tablet bd8ab27870b84185b6835c8ea1c461fd (table test-table [id=488deff3ec964ad69b8eca0bee4ac123]) successfully deleted
Aug 06 09:23:55 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:23:55.997501 1340 ts_tablet_manager.cc:1918] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:23:55.997718 1340 log.cc:1196] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/bd8ab27870b84185b6835c8ea1c461fd
I20260806 09:23:55.998494 1340 ts_tablet_manager.cc:1939] T bd8ab27870b84185b6835c8ea1c461fd P cf102ef35bfa4832a056354b052becfa: Deleting consensus metadata
I20260806 09:23:56.000645 505 catalog_manager.cc:4841] TS cf102ef35bfa4832a056354b052becfa (127.16.86.129:38633): tablet bd8ab27870b84185b6835c8ea1c461fd (table test-table [id=488deff3ec964ad69b8eca0bee4ac123]) successfully deleted
I20260806 09:23:56.003530 1339 ts_tablet_manager.cc:1918] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:23:56.003813 1339 log.cc:1196] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/bd8ab27870b84185b6835c8ea1c461fd
I20260806 09:23:56.004580 1339 ts_tablet_manager.cc:1939] T bd8ab27870b84185b6835c8ea1c461fd P d6a1acc0aa5b4f07bb2335217635a49f: Deleting consensus metadata
I20260806 09:23:56.007544 503 catalog_manager.cc:4841] TS d6a1acc0aa5b4f07bb2335217635a49f (127.16.86.130:36139): tablet bd8ab27870b84185b6835c8ea1c461fd (table test-table [id=488deff3ec964ad69b8eca0bee4ac123]) successfully deleted
I20260806 09:23:56.012703 1300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:55.963237 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.16.86.129:38633)
0806 09:23:55.963540 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:55.963550 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:55.974456 (+ 10906us) server_negotiation.cc:407] Connection header received
0806 09:23:55.975164 (+ 708us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:55.975174 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:55.975305 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:55.975410 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:23:55.978478 (+ 3068us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.979890 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.984058 (+ 4168us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:55.984717 (+ 659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:55.984824 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:56.000301 (+ 15477us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:23:56.000334 (+ 33us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:23:56.000381 (+ 47us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:23:56.000510 (+ 129us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:23:56.006835 (+ 6325us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:56.009230 (+ 2395us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:56.009240 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:56.009243 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:56.009509 (+ 266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:56.010266 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:56.010271 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:56.010294 (+ 23us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:56.010964 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:23:56.011583 (+ 619us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:56.011994 (+ 411us) server_negotiation.cc:299] Negotiation successful
0806 09:23:56.012236 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:23:56.013787 847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:23:56.113849 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 649
I20260806 09:23:56.153541 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 860
Aug 06 09:23:56 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:23:56.174876 1301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:56.154732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51402 (local address 127.16.86.130:36139)
0806 09:23:56.155043 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:56.155050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:56.155081 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:23:56.155176 (+ 95us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:56.155183 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:56.155289 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:56.155449 (+ 160us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:23:56.157279 (+ 1830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:56.158472 (+ 1193us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:56.160004 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:56.160743 (+ 739us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:56.160890 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:56.166170 (+ 5280us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:23:56.166195 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:23:56.166224 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:23:56.166282 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:23:56.170852 (+ 4570us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:56.171780 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:56.171800 (+ 20us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:56.171859 (+ 59us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:56.172086 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:56.172540 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:56.172561 (+ 21us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:56.172565 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:56.173193 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:23:56.173619 (+ 426us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:56.173987 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 09:23:56.174362 (+ 375us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260806 09:23:56.175985 1056 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:23:56.271808 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 860
I20260806 09:23:56.298928 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 1072
Aug 06 09:23:56 dist-test-slave-h3ft krb5kdc[32278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:23:56.317611 1306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:23:56.299894 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51780 (local address 127.16.86.131:34469)
0806 09:23:56.300170 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 09:23:56.300176 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:23:56.300198 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:23:56.300311 (+ 113us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:23:56.300316 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:23:56.300396 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:23:56.300513 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:23:56.301930 (+ 1417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:56.302801 (+ 871us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:56.304204 (+ 1403us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:23:56.304711 (+ 507us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:23:56.304872 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:23:56.310377 (+ 5505us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:23:56.310397 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:23:56.310422 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:23:56.310487 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:23:56.314509 (+ 4022us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:56.315319 (+ 810us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:56.315326 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:56.315329 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:56.315514 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:23:56.315929 (+ 415us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:23:56.315933 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:23:56.315935 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:23:56.316301 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:23:56.316658 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0806 09:23:56.316977 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 09:23:56.317248 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260806 09:23:56.318686 1268 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:23:56.446019 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 1072
I20260806 09:23:56.473556 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 479
I20260806 09:23:56.475349 573 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:23:56.598436 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 479
I20260806 09:23:56.637730 16730 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 09:23:56.998180 16730 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 09:23:56.998620 16730 mini_ranger.cc:66] Stopping Ranger...
I20260806 09:23:57.357461 16730 mini_ranger.cc:68] Stopped Ranger
2026-08-06 09:23:57.357 UTC [32328] LOG: received smart shutdown request
2026-08-06 09:23:57.363 UTC [32328] LOG: background worker "logical replication launcher" (PID 32340) exited with exit code 1
2026-08-06 09:23:57.363 UTC [32334] LOG: shutting down
2026-08-06 09:23:57.590 UTC [32328] LOG: database system is shut down
2026-08-06T09:23:57Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (58407 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:23:57 dist-test-slave-h3ft krb5kdc[1361](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:23:57 dist-test-slave-h3ft krb5kdc[1361](info): set up 2 sockets
Aug 06 09:23:57 dist-test-slave-h3ft krb5kdc[1361](info): commencing operation
krb5kdc: starting...
W20260806 09:23:59.133353 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.340s user 0.000s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:23:59 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008239, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:23:59Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:23:59Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.16.86.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.16.86.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/rangeradmin_127.16.86.148@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/rangeradmin_127.16.86.148@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.16.86.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.16.86.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/rangerlookup_127.16.86.148@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/rangerlookup_127.16.86.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.86.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.16.86.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/HTTP_127.16.86.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/HTTP_127.16.86.148@KRBTEST.COM.keytab.
I20260806 09:23:59.647078 16730 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default max_connections ... 100
selecting default shared_buffers ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
You can change this by editing pg_hba.conf or using the option -A, or
--auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskKAHevx/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/postgres -l logfile start
2026-08-06 09:24:03.106 UTC [1412] LOG: starting PostgreSQL 12.2 on x86_64-pc-linux-gnu, compiled by cc (GCC) 8.3.1 20190311 (Red Hat 8.3.1-3), 64-bit
2026-08-06 09:24:03.106 UTC [1412] LOG: listening on IPv4 address "127.16.86.148", port 48695
2026-08-06 09:24:03.111 UTC [1412] LOG: listening on Unix socket "/tmp/.s.PGSQL.48695"
2026-08-06 09:24:03.139 UTC [1414] LOG: database system was shut down at 2026-08-06 09:24:00 UTC
2026-08-06 09:24:03.145 UTC [1412] LOG: database system is ready to accept connections
I20260806 09:24:04.352169 16730 mini_postgres.cc:96] Postgres bound to 48695
2026-08-06 09:24:04.382 UTC [1425] FATAL: database "slave" does not exist
127.16.86.148:48695 - accepting connections
I20260806 09:24:04.383606 16730 mini_ranger.cc:161] Starting Ranger...
I20260806 09:24:04.425917 16730 mini_ranger.cc:84] Created miniranger Postgres user
I20260806 09:24:05.114696 16730 mini_ranger.cc:87] Created ranger Postgres database
I20260806 09:24:05.114986 16730 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-admin
2026-08-06 09:24:05,456 [I] DB FLAVOR :POSTGRES
2026-08-06 09:24:05,456 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 09:24:05,456 [I] Checking connection..
2026-08-06 09:24:05,457 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2026-08-06 09:24:05,785 [I] Checking connection passed.
2026-08-06 09:24:05,785 [I] --------- Verifying version history table ---------
2026-08-06 09:24:05,785 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:24:06,111 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 09:24:06,112 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:24:06,428 [I] Table x_db_version_h does not exist in database ranger
2026-08-06 09:24:06,428 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2026-08-06 09:24:06,428 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2026-08-06 09:24:06.726 UTC [1513] WARNING: there is no transaction in progress
2026-08-06 09:24:06,736 [I] create_dbversion_catalog.sql file imported successfully
2026-08-06 09:24:06,736 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2026-08-06 09:24:07,062 [I] Table x_db_version_h already exists in database 'ranger'
2026-08-06 09:24:07,063 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 09:24:07,063 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2026-08-06 09:24:07,367 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2026-08-06 09:24:07,641 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-h3ft.c.gcp-upstream.internal','N') ;"
2026-08-06 09:24:07,932 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 09:24:08,210 [I] Table x_portal_user does not exist in database ranger
2026-08-06 09:24:08,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 09:24:08,495 [I] Table x_policy_ref_group does not exist in database ranger
2026-08-06 09:24:08,495 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 09:24:08,776 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 09:24:09,057 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2026-08-06 09:24:09,058 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2026-08-06 09:24:10.200 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.219 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.239 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.257 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.275 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.297 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.361 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.371 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.387 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.398 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.411 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.428 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.440 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.447 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.456 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:10.463 UTC [1664] WARNING: there is no transaction in progress
2026-08-06 09:24:11,245 [I] ranger_core_db_postgres.sql file imported successfully
2026-08-06 09:24:11,245 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2026-08-06 09:24:11,521 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2026-08-06 09:24:11,521 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2026-08-06 09:24:11,796 [I] Table x_portal_user already exists in database 'ranger'
2026-08-06 09:24:11,797 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2026-08-06 09:24:12,089 [I] Table x_policy_ref_group already exists in database 'ranger'
2026-08-06 09:24:12,090 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2026-08-06 09:24:12,377 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2026-08-06 09:24:12,659 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-h3ft.c.gcp-upstream.internal';"
2026-08-06 09:24:12,937 [I] CORE_DB_SCHEMA import status has been updated
2026-08-06 09:24:12,937 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2026-08-06 09:24:13,228 [I] DB_PATCHES have already been applied
I20260806 09:24:13.235963 16730 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-admin:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskKAHevx/thirdparty/src/hadoop-3.2.0/*:/tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20260806 09:24:13.236109 16730 mini_ranger.cc:193] Using host: 127.16.86.148
I20260806 09:24:13.263660 16730 mini_ranger.cc:227] Ranger admin URL: http://127.16.86.148:41661
Aug 06, 2026 9:24:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 9:24:13 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 9:24:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-admin/ews/webapp
Aug 06, 2026 9:24:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Aug 06, 2026 9:24:13 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-admin/ews/webapp] .....
Aug 06, 2026 9:24:14 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-admin/ews/webapp].
Aug 06, 2026 9:24:14 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
I20260806 09:24:14.652171 16730 mini_ranger.cc:160] Time spent starting Ranger: real 10.269s user 0.001s sys 0.085s
Aug 06, 2026 9:24:37 AM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Aug 06, 2026 9:24:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Aug 06, 2026 9:24:37 AM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Aug 06, 2026 9:24:37 AM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Aug 06, 2026 9:24:37 AM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Aug 06, 2026 9:24:38 AM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
I20260806 09:24:39.224185 16730 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.16.86.148@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.16.86.148@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/rangerkms_127.16.86.148@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.16.86.148@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/rangerkms_127.16.86.148@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.16.86.148@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.16.86.148@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.16.86.148@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/HTTP_127.16.86.148@KRBTEST.COM.keytab.
Entry for principal HTTP/127.16.86.148@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/HTTP_127.16.86.148@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Aug 06 09:24:39 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008279, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20260806 09:24:39.498857 16730 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20260806 09:24:39.532949 16730 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20260806 09:24:40.650349 16730 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20260806 09:24:40.650490 16730 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-kms
I20260806 09:24:40.650529 16730 mini_ranger_kms.cc:227] Using postgres at 127.16.86.148:48695
2026-08-06 09:24:41,121 [I] DB FLAVOR :POSTGRES
2026-08-06 09:24:41,122 [I] --------- Verifying Ranger DB connection ---------
2026-08-06 09:24:41,122 [I] Checking connection
2026-08-06 09:24:41,122 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2026-08-06 09:24:41,416 [I] connection success
2026-08-06 09:24:41,417 [I] --------- Verifying Ranger DB tables ---------
2026-08-06 09:24:41,417 [I] Verifying table ranger_masterkey in database rangerkms
2026-08-06 09:24:41,417 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2026-08-06 09:24:41,699 [I] Table ranger_masterkey does not exist in database rangerkms
2026-08-06 09:24:41,699 [I] --------- Importing Ranger Core DB Schema ---------
2026-08-06 09:24:41,700 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2026-08-06 09:24:41,700 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.16.86.148:48695/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2026-08-06 09:24:42,007 [I] kms_core_db_postgres.sql DB schema imported successfully
I20260806 09:24:42.579957 16730 mini_ranger_kms.cc:310] Created kms service
I20260806 09:24:42.791879 16730 mini_ranger_kms.cc:326] Created kudu user
I20260806 09:24:42.847348 16730 mini_ranger_kms.cc:343] Created rangerkms user
Aug 06 09:24:42 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008279, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:24:43.400117 16730 mini_ranger_kms.cc:384] Added ranger policy
I20260806 09:24:43.400511 16730 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-kms:/tmp/dist-test-taskKAHevx/build/asan/bin/postgresql.jar:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskKAHevx/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20260806 09:24:43.400588 16730 mini_ranger_kms.cc:240] Using host: 127.16.86.148
I20260806 09:24:43.427906 16730 mini_ranger_kms.cc:276] Ranger KMS PID: 1922
I20260806 09:24:43.428078 16730 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.16.86.148:58699
Aug 06, 2026 9:24:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Aug 06, 2026 9:24:44 AM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Aug 06, 2026 9:24:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Aug 06, 2026 9:24:44 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-kms/ews/webapp] .....
Aug 06, 2026 9:24:45 AM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-kms/ews/webapp].
Aug 06, 2026 9:24:45 AM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Aug 06 09:24:46 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786008286, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.86.148@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:24:46.346022 16730 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 6.847s user 0.079s sys 0.442s
I20260806 09:24:46.346347 16730 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20260806 09:24:46.346558 16730 mini_ranger_kms.cc:401] 127.16.86.148:58699/kms/v1/keys
Aug 06 09:24:46 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008286, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.86.148@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
Aug 06 09:24:47 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008286, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.16.86.148@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
Aug 06 09:24:49 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:24:49.877668 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:42405
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:42405
--ranger_config_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.86.148:58699/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:24:50.328864 2003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:24:50.329277 2003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:24:50.329456 2003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:24:50.337637 2003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:24:50.337847 2003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:24:50.337965 2003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:24:50.338064 2003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:24:50.338124 2003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:24:50.350029 2003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/wal
--ranger_kms_url=127.16.86.148:58699/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:42405
--ranger_config_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:24:50.351771 2003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:24:50.353453 2003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:24:50.368727 2012 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:50.368582 2013 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:50.368844 2016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:24:50.369808 2003 server_base.cc:1034] running on GCE node
I20260806 09:24:50.370805 2003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:24:50.372604 2003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:24:50.373838 2003 hybrid_clock.cc:648] HybridClock initialized: now 1786008290373776 us; error 51 us; skew 500 ppm
I20260806 09:24:50.374287 2003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:24:50 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008290, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:24:50.378062 2003 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:24:50.379999 2003 webserver.cc:460] Webserver started at http://127.16.86.190:44889/ using document root <none> and password file <none>
I20260806 09:24:50.380764 2003 fs_manager.cc:359] Metadata directory not provided
I20260806 09:24:50.380914 2003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:24:50.381265 2003 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:24:50 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008290, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:24:50.453459 2003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "1a0dd92539c940c68d966a15186817d4"
format_stamp: "Formatted at 2026-08-06 09:24:50 on dist-test-slave-h3ft"
server_key: "2214d484d54b30d1007897526ae3a156"
server_key_iv: "9234e21325bc6c3f9854ad34860b0f92"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:50.454720 2003 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "1a0dd92539c940c68d966a15186817d4"
format_stamp: "Formatted at 2026-08-06 09:24:50 on dist-test-slave-h3ft"
server_key: "2214d484d54b30d1007897526ae3a156"
server_key_iv: "9234e21325bc6c3f9854ad34860b0f92"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:50.462172 2003 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260806 09:24:50.490744 2026 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:50.493052 2003 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.002s sys 0.001s
I20260806 09:24:50.493343 2003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "1a0dd92539c940c68d966a15186817d4"
format_stamp: "Formatted at 2026-08-06 09:24:50 on dist-test-slave-h3ft"
server_key: "2214d484d54b30d1007897526ae3a156"
server_key_iv: "9234e21325bc6c3f9854ad34860b0f92"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:50.493644 2003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:24:50.530244 2003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:24:50.535526 2003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:24:50.535925 2003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:24:50.560585 2003 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:42405
I20260806 09:24:50.560650 2107 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:42405 every 8 connection(s)
I20260806 09:24:50.562739 2003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:24:50.567422 2109 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:24:50.568945 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 2003
I20260806 09:24:50.569217 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:24:50.570010 16730 external_mini_cluster.cc:1428] Setting key 083efeaeff611afb2a52bd7840c98b7c
I20260806 09:24:50.581705 2109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4: Bootstrap starting.
I20260806 09:24:50.585542 2109 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4: Neither blocks nor log segments found. Creating new log.
I20260806 09:24:50.586612 2109 log.cc:824] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:24:50 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:24:50.590782 2109 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4: No bootstrap required, opened a new log
I20260806 09:24:50.600085 2114 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:50.572830 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60970 (local address 127.16.86.190:42405)
0806 09:24:50.573884 (+ 1054us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:50.573903 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:50.573969 (+ 66us) server_negotiation.cc:407] Connection header received
0806 09:24:50.576521 (+ 2552us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:50.576550 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:50.577077 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:50.577572 (+ 495us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:50.579376 (+ 1804us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:50.581317 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:50.583045 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:50.584151 (+ 1106us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:50.584353 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:50.590187 (+ 5834us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:50.590199 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:50.590224 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:50.590278 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:50.594233 (+ 3955us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:50.595415 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:50.595424 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:50.595434 (+ 10us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:50.595678 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:50.596179 (+ 501us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:50.596183 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:50.596185 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:50.597007 (+ 822us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:50.597290 (+ 283us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:50.598013 (+ 723us) server_negotiation.cc:299] Negotiation successful
0806 09:24:50.598523 (+ 510us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":721,"thread_start_us":425,"threads_started":1}
I20260806 09:24:50.600862 2109 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0dd92539c940c68d966a15186817d4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42405 } }
I20260806 09:24:50.601318 2109 raft_consensus.cc:374] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:24:50.601451 2109 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a0dd92539c940c68d966a15186817d4, State: Initialized, Role: FOLLOWER
I20260806 09:24:50.602186 2109 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0dd92539c940c68d966a15186817d4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42405 } }
I20260806 09:24:50.602516 2109 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:24:50.602698 2109 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:24:50.602908 2109 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:24:50.605655 2109 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0dd92539c940c68d966a15186817d4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42405 } }
I20260806 09:24:50.606149 2109 leader_election.cc:302] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a0dd92539c940c68d966a15186817d4; no voters:
I20260806 09:24:50.606863 2109 leader_election.cc:288] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:24:50.607255 2117 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:24:50.608335 2117 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 1 LEADER]: Becoming Leader. State: Replica: 1a0dd92539c940c68d966a15186817d4, State: Running, Role: LEADER
I20260806 09:24:50.609128 2117 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0dd92539c940c68d966a15186817d4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42405 } }
I20260806 09:24:50.609476 2109 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:24:50.615535 2119 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a0dd92539c940c68d966a15186817d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0dd92539c940c68d966a15186817d4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42405 } } }
I20260806 09:24:50.616099 2119 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [sys.catalog]: This master's current role is: LEADER
I20260806 09:24:50.616839 2122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a0dd92539c940c68d966a15186817d4. Latest consensus state: current_term: 1 leader_uuid: "1a0dd92539c940c68d966a15186817d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0dd92539c940c68d966a15186817d4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42405 } } }
I20260806 09:24:50.617127 2122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [sys.catalog]: This master's current role is: LEADER
I20260806 09:24:50.617189 2131 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:24:50.623801 2131 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:24:50.624799 2109 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20260806 09:24:50.650568 2131 catalog_manager.cc:1347] Generated new cluster ID: 1d57ac99a1ef4a778e7a1988683628e3
I20260806 09:24:50.651526 2131 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:24:50.678206 2131 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:24:50.679888 2131 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:24:50.701261 2131 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4: Generated new TSK 0
I20260806 09:24:50.702313 2131 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:24:51.550217 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.86.148:58699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42405
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:24:51 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786008291, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20260806 09:24:52.037029 2181 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:24:52.037717 2181 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:24:52.037943 2181 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:24:52.046785 2181 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:24:52.047061 2181 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:24:52.047312 2181 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:24:52.060932 2181 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.16.86.148:58699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:24:52.062992 2181 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:24:52.064852 2181 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:24:52.086831 2195 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:52.087090 2197 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:52.096508 2203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:24:52.097457 2181 server_base.cc:1034] running on GCE node
I20260806 09:24:52.098713 2181 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:24:52.116504 2181 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:24:52.120239 2181 hybrid_clock.cc:648] HybridClock initialized: now 1786008292120107 us; error 132 us; skew 500 ppm
I20260806 09:24:52.120962 2181 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:24:52 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008292, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:24:52.126199 2181 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:24:52.138659 2181 webserver.cc:460] Webserver started at http://127.16.86.129:38705/ using document root <none> and password file <none>
I20260806 09:24:52.139895 2181 fs_manager.cc:359] Metadata directory not provided
I20260806 09:24:52.140113 2181 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:24:52.140554 2181 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:24:52 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008292, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:24:52.167361 2181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "b2d4de180c664567bc30ce07c92dfb8e"
format_stamp: "Formatted at 2026-08-06 09:24:52 on dist-test-slave-h3ft"
server_key: "97636af8fef18369845b44b2e6b518ee"
server_key_iv: "7e90a4da7619ed27ad4d33e3da290e10"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:52.169068 2181 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "b2d4de180c664567bc30ce07c92dfb8e"
format_stamp: "Formatted at 2026-08-06 09:24:52 on dist-test-slave-h3ft"
server_key: "97636af8fef18369845b44b2e6b518ee"
server_key_iv: "7e90a4da7619ed27ad4d33e3da290e10"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:52.197947 2181 fs_manager.cc:691] Time spent creating directory manager: real 0.028s user 0.007s sys 0.007s
I20260806 09:24:52.239789 2212 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:52.242115 2181 fs_manager.cc:725] Time spent opening block manager: real 0.020s user 0.011s sys 0.008s
I20260806 09:24:52.242434 2181 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "b2d4de180c664567bc30ce07c92dfb8e"
format_stamp: "Formatted at 2026-08-06 09:24:52 on dist-test-slave-h3ft"
server_key: "97636af8fef18369845b44b2e6b518ee"
server_key_iv: "7e90a4da7619ed27ad4d33e3da290e10"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:52.242986 2181 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:24:52.305989 2181 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:24:52.312106 2181 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:24:52.312592 2181 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:24:52.320267 2181 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:24:52.328706 2181 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:24:52.328900 2181 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 09:24:52.329115 2181 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:24:52.329233 2181 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Aug 06 09:24:52 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008291, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:24:52.692322 2181 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:44223
I20260806 09:24:52.692358 2381 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:44223 every 8 connection(s)
I20260806 09:24:52.694433 2181 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:24:52.698115 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 2181
I20260806 09:24:52.698378 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:24:52.699659 16730 external_mini_cluster.cc:1428] Setting key bd4940d2d4dba943ae716e98cc9f32c4
Aug 06 09:24:52 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008292, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:24:52.727442 2385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:52.698301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:44783 (local address 127.16.86.190:42405)
0806 09:24:52.707551 (+ 9250us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:52.707556 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:52.707578 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:24:52.707755 (+ 177us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:52.707761 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:52.707863 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:52.708019 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:52.710034 (+ 2015us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:52.711143 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:52.712999 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:52.713477 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:52.713647 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:52.718030 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:52.718037 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:52.718040 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:52.718093 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:52.722133 (+ 4040us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:52.723488 (+ 1355us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:52.723497 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:52.723501 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:52.723806 (+ 305us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:52.724550 (+ 744us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:52.724555 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:52.724559 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:52.725009 (+ 450us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:52.725211 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:52.726460 (+ 1249us) server_negotiation.cc:299] Negotiation successful
0806 09:24:52.726825 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9096,"thread_start_us":8931,"threads_started":1}
I20260806 09:24:52.729012 2386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:52.699090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:42405 (local address 127.16.86.129:44783)
0806 09:24:52.702050 (+ 2960us) negotiation.cc:107] Waiting for socket to connect
0806 09:24:52.702141 (+ 91us) client_negotiation.cc:174] Beginning negotiation
0806 09:24:52.706649 (+ 4508us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:24:52.708349 (+ 1700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:24:52.708359 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:24:52.708989 (+ 630us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:24:52.709825 (+ 836us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:52.709848 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:52.711345 (+ 1497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:52.711350 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:52.712820 (+ 1470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:52.712835 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:52.713788 (+ 953us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:52.713796 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:52.714076 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:52.714942 (+ 866us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:24:52.714974 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:24:52.717732 (+ 2758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:24:52.722342 (+ 4610us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:24:52.722357 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:24:52.722381 (+ 24us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:24:52.723202 (+ 821us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:24:52.724022 (+ 820us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:24:52.724031 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:24:52.724038 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:24:52.724361 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:24:52.725203 (+ 842us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:24:52.725215 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:24:52.725856 (+ 641us) client_negotiation.cc:769] Sending connection context
0806 09:24:52.726274 (+ 418us) client_negotiation.cc:240] Negotiation successful
0806 09:24:52.726958 (+ 684us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2413,"thread_start_us":2040,"threads_started":1}
I20260806 09:24:52.731627 2382 heartbeater.cc:347] Connected to a master server at 127.16.86.190:42405
I20260806 09:24:52.732275 2382 heartbeater.cc:464] Registering TS with master...
I20260806 09:24:52.733717 2382 heartbeater.cc:511] Master 127.16.86.190:42405 requested a full tablet report, sending...
I20260806 09:24:52.737536 2056 ts_manager.cc:194] Registered new tserver with Master: b2d4de180c664567bc30ce07c92dfb8e (127.16.86.129:44223)
I20260806 09:24:52.740091 2056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:44783
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:24:52.800181 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.86.148:58699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42405
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:24:52 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008291, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
W20260806 09:24:53.386946 2392 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:24:53.387429 2392 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:24:53.387542 2392 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:24:53.395998 2392 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:24:53.396157 2392 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:24:53.396287 2392 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:24:53.407855 2392 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.16.86.148:58699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:24:53.409672 2392 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:24:53.411504 2392 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:24:53.429008 2409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:53.429328 2410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:53.430321 2414 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:24:53.430990 2392 server_base.cc:1034] running on GCE node
I20260806 09:24:53.431838 2392 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:24:53.433137 2392 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:24:53.434474 2392 hybrid_clock.cc:648] HybridClock initialized: now 1786008293434400 us; error 56 us; skew 500 ppm
I20260806 09:24:53.434929 2392 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:24:53 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008293, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:24:53.438906 2392 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:24:53.440860 2392 webserver.cc:460] Webserver started at http://127.16.86.130:34767/ using document root <none> and password file <none>
I20260806 09:24:53.441754 2392 fs_manager.cc:359] Metadata directory not provided
I20260806 09:24:53.441923 2392 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:24:53.442333 2392 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:24:53 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008293, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:24:53.461999 2392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "31548b00d49645b6baf237b840061801"
format_stamp: "Formatted at 2026-08-06 09:24:53 on dist-test-slave-h3ft"
server_key: "94a4c1b957b746931d44c92abb014c8e"
server_key_iv: "47a50bfaf868b8c8f88209dd4d5e2ba5"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:53.463307 2392 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "31548b00d49645b6baf237b840061801"
format_stamp: "Formatted at 2026-08-06 09:24:53 on dist-test-slave-h3ft"
server_key: "94a4c1b957b746931d44c92abb014c8e"
server_key_iv: "47a50bfaf868b8c8f88209dd4d5e2ba5"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:53.471544 2392 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260806 09:24:53.493443 2425 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:53.495800 2392 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:24:53.496084 2392 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "31548b00d49645b6baf237b840061801"
format_stamp: "Formatted at 2026-08-06 09:24:53 on dist-test-slave-h3ft"
server_key: "94a4c1b957b746931d44c92abb014c8e"
server_key_iv: "47a50bfaf868b8c8f88209dd4d5e2ba5"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:53.496405 2392 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:24:53.531361 2392 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:24:53.537390 2392 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:24:53.537829 2392 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:24:53.539273 2392 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:24:53.541330 2392 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:24:53.541448 2392 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:53.541625 2392 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:24:53.541710 2392 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:53.590958 2392 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:37083
I20260806 09:24:53.591133 2600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:37083 every 8 connection(s)
I20260806 09:24:53.592932 2392 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:24:53.599790 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 2392
I20260806 09:24:53.600013 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:24:53.600592 16730 external_mini_cluster.cc:1428] Setting key be8eeb937d9d6cb9376ee300912b66a4
Aug 06 09:24:53 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008293, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:24:53.621148 2604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:53.596554 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:49785 (local address 127.16.86.190:42405)
0806 09:24:53.597178 (+ 624us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:53.597183 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:53.598737 (+ 1554us) server_negotiation.cc:407] Connection header received
0806 09:24:53.601586 (+ 2849us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:53.601591 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:53.601700 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:53.601833 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:53.603988 (+ 2155us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:53.604893 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:53.607690 (+ 2797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:53.608285 (+ 595us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:53.608471 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:53.613702 (+ 5231us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:53.613709 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:53.613712 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:53.613765 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:53.617200 (+ 3435us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:53.618242 (+ 1042us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:53.618247 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:53.618250 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:53.618406 (+ 156us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:53.618936 (+ 530us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:53.618940 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:53.618943 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:53.619357 (+ 414us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:53.619551 (+ 194us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:53.620436 (+ 885us) server_negotiation.cc:299] Negotiation successful
0806 09:24:53.620701 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":472,"thread_start_us":301,"threads_started":1}
I20260806 09:24:53.622465 2605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:53.597383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:42405 (local address 127.16.86.130:49785)
0806 09:24:53.598481 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
0806 09:24:53.598540 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 09:24:53.601237 (+ 2697us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:24:53.602174 (+ 937us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:24:53.602189 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:24:53.602884 (+ 695us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:24:53.603741 (+ 857us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:53.603769 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:53.605106 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:53.605115 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:53.607458 (+ 2343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:53.607484 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:53.608625 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:53.608636 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:53.609014 (+ 378us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:53.609973 (+ 959us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:24:53.610011 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:24:53.613335 (+ 3324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:24:53.617440 (+ 4105us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:24:53.617451 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:24:53.617472 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:24:53.618071 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:24:53.618533 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:24:53.618538 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:24:53.618542 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:24:53.618791 (+ 249us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:24:53.619514 (+ 723us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:24:53.619524 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:24:53.619974 (+ 450us) client_negotiation.cc:769] Sending connection context
0806 09:24:53.620317 (+ 343us) client_negotiation.cc:240] Negotiation successful
0806 09:24:53.620985 (+ 668us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":772,"thread_start_us":362,"threads_started":1}
I20260806 09:24:53.624534 2601 heartbeater.cc:347] Connected to a master server at 127.16.86.190:42405
I20260806 09:24:53.624986 2601 heartbeater.cc:464] Registering TS with master...
I20260806 09:24:53.626047 2601 heartbeater.cc:511] Master 127.16.86.190:42405 requested a full tablet report, sending...
I20260806 09:24:53.628250 2056 ts_manager.cc:194] Registered new tserver with Master: 31548b00d49645b6baf237b840061801 (127.16.86.130:37083)
I20260806 09:24:53.629608 2056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:49785
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:24:53.679867 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.86.148:58699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42405
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:24:53.744912 2382 heartbeater.cc:503] Master 127.16.86.190:42405 was elected leader, sending a full tablet report...
W20260806 09:24:54.094939 2611 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:24:54.095383 2611 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:24:54.095549 2611 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:24:54.104414 2611 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:24:54.104586 2611 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:24:54.104722 2611 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:24:54.116724 2611 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.16.86.148:58699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:24:54.118664 2611 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:24:54.120407 2611 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:24:54.137107 2621 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:54.137363 2623 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:54.137254 2627 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:24:54.138758 2611 server_base.cc:1034] running on GCE node
I20260806 09:24:54.139711 2611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:24:54.141002 2611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:24:54.142306 2611 hybrid_clock.cc:648] HybridClock initialized: now 1786008294142246 us; error 47 us; skew 500 ppm
I20260806 09:24:54.142787 2611 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:24:54 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008294, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:24:54.147156 2611 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:24:54.149403 2611 webserver.cc:460] Webserver started at http://127.16.86.131:46545/ using document root <none> and password file <none>
I20260806 09:24:54.150365 2611 fs_manager.cc:359] Metadata directory not provided
I20260806 09:24:54.150543 2611 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:24:54.150950 2611 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
Aug 06 09:24:54 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008294, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:24:54.170743 2611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "88b022611a4f4cb5a5fead1585ebfc88"
format_stamp: "Formatted at 2026-08-06 09:24:54 on dist-test-slave-h3ft"
server_key: "b28ef32c54e36210af4158ad66aaaff7"
server_key_iv: "d224ab15ce4d4edd714797a5975efb12"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:54.171763 2611 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "88b022611a4f4cb5a5fead1585ebfc88"
format_stamp: "Formatted at 2026-08-06 09:24:54 on dist-test-slave-h3ft"
server_key: "b28ef32c54e36210af4158ad66aaaff7"
server_key_iv: "d224ab15ce4d4edd714797a5975efb12"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:54.179091 2611 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260806 09:24:54.195940 2636 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:54.198287 2611 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260806 09:24:54.198621 2611 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "88b022611a4f4cb5a5fead1585ebfc88"
format_stamp: "Formatted at 2026-08-06 09:24:54 on dist-test-slave-h3ft"
server_key: "b28ef32c54e36210af4158ad66aaaff7"
server_key_iv: "d224ab15ce4d4edd714797a5975efb12"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:54.198916 2611 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:24:54.243583 2611 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:24:54.247969 2611 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:24:54.248317 2611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:24:54.249677 2611 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:24:54.251488 2611 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:24:54.251608 2611 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:54.251765 2611 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:24:54.251842 2611 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:54.294800 2611 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:39453
I20260806 09:24:54.294906 2800 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:39453 every 8 connection(s)
I20260806 09:24:54.296304 2611 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:24:54.301192 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 2611
I20260806 09:24:54.301421 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:24:54.302080 16730 external_mini_cluster.cc:1428] Setting key 98a4d9067ec9483a856b72874c8085dd
Aug 06 09:24:54 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008294, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:24:54.322511 2806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:54.299964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:57965 (local address 127.16.86.190:42405)
0806 09:24:54.300696 (+ 732us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:54.300701 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:54.302273 (+ 1572us) server_negotiation.cc:407] Connection header received
0806 09:24:54.305171 (+ 2898us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:54.305177 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:54.305294 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:54.305443 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:54.307426 (+ 1983us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:54.308325 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:54.310101 (+ 1776us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:54.310747 (+ 646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:54.310898 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:54.314938 (+ 4040us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:54.314944 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:54.314948 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:54.314986 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:54.318893 (+ 3907us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:54.320084 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:54.320088 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:54.320091 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:54.320243 (+ 152us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:54.320716 (+ 473us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:54.320719 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:54.320721 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:54.321032 (+ 311us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:54.321183 (+ 151us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:54.321977 (+ 794us) server_negotiation.cc:299] Negotiation successful
0806 09:24:54.322191 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":586,"thread_start_us":311,"threads_started":1}
I20260806 09:24:54.323549 2807 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:54.300781 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:42405 (local address 127.16.86.131:57965)
0806 09:24:54.301974 (+ 1193us) negotiation.cc:107] Waiting for socket to connect
0806 09:24:54.302025 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:24:54.304769 (+ 2744us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:24:54.305763 (+ 994us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:24:54.305774 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:24:54.306348 (+ 574us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:24:54.307222 (+ 874us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:54.307247 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:54.308508 (+ 1261us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:54.308513 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:54.309941 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:54.309956 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:54.311082 (+ 1126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:54.311092 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:54.311398 (+ 306us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:54.312189 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:24:54.312214 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:24:54.314667 (+ 2453us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:24:54.319197 (+ 4530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:24:54.319211 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:24:54.319259 (+ 48us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:24:54.319918 (+ 659us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:24:54.320357 (+ 439us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:24:54.320360 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:24:54.320363 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:24:54.320596 (+ 233us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:24:54.321169 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:24:54.321176 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:24:54.321578 (+ 402us) client_negotiation.cc:769] Sending connection context
0806 09:24:54.321835 (+ 257us) client_negotiation.cc:240] Negotiation successful
0806 09:24:54.322286 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":837,"thread_start_us":436,"threads_started":1}
I20260806 09:24:54.325086 2801 heartbeater.cc:347] Connected to a master server at 127.16.86.190:42405
I20260806 09:24:54.325558 2801 heartbeater.cc:464] Registering TS with master...
I20260806 09:24:54.326519 2801 heartbeater.cc:511] Master 127.16.86.190:42405 requested a full tablet report, sending...
I20260806 09:24:54.328401 2056 ts_manager.cc:194] Registered new tserver with Master: 88b022611a4f4cb5a5fead1585ebfc88 (127.16.86.131:39453)
I20260806 09:24:54.329381 2056 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:57965
I20260806 09:24:54.332365 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:24:54.334266 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 2181
Aug 06 09:24:54 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:24:54.354535 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:54.335518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51802 (local address 127.16.86.129:44223)
0806 09:24:54.336222 (+ 704us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:54.336235 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:54.336265 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:24:54.336333 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:54.336339 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:54.336565 (+ 226us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:54.336714 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:54.338555 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:54.339771 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:54.341597 (+ 1826us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:54.342251 (+ 654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:54.342378 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:54.347590 (+ 5212us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:54.347606 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:54.347624 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:54.347675 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:54.351317 (+ 3642us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:54.352262 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:54.352270 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:54.352272 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:54.352451 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:54.352915 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:54.352919 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:54.352921 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:54.353314 (+ 393us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:54.353634 (+ 320us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:54.353907 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 09:24:54.354139 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":522,"thread_start_us":341,"threads_started":1}
I20260806 09:24:54.356083 2376 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:24:54.445115 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 2181
I20260806 09:24:54.478183 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 2392
Aug 06 09:24:54 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:24:54.498111 2812 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:54.479412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52164 (local address 127.16.86.130:37083)
0806 09:24:54.480066 (+ 654us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:54.480073 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:54.480110 (+ 37us) server_negotiation.cc:407] Connection header received
0806 09:24:54.480180 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:54.480187 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:54.480409 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:54.480556 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:54.482177 (+ 1621us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:54.483506 (+ 1329us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:54.485173 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:54.485842 (+ 669us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:54.485969 (+ 127us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:54.491566 (+ 5597us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:54.491581 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:54.491597 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:54.491646 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:54.494837 (+ 3191us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:54.495756 (+ 919us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:54.495763 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:54.495766 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:54.495902 (+ 136us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:54.496330 (+ 428us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:54.496334 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:54.496336 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:54.496717 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:54.497073 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:54.497439 (+ 366us) server_negotiation.cc:299] Negotiation successful
0806 09:24:54.497730 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":473,"thread_start_us":358,"threads_started":1}
I20260806 09:24:54.499958 2594 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:24:54.588888 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 2392
I20260806 09:24:54.615965 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 2611
Aug 06 09:24:54 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008289, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:24:54.636761 2816 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:54.617370 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49122 (local address 127.16.86.131:39453)
0806 09:24:54.618205 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:54.618213 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:54.618249 (+ 36us) server_negotiation.cc:407] Connection header received
0806 09:24:54.618315 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:54.618322 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:54.618550 (+ 228us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:54.618696 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:54.620686 (+ 1990us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:54.621808 (+ 1122us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:54.623694 (+ 1886us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:54.624293 (+ 599us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:54.624409 (+ 116us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:54.630113 (+ 5704us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:54.630126 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:54.630143 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:54.630202 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:54.633462 (+ 3260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:54.634308 (+ 846us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:54.634315 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:54.634318 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:54.634464 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:54.634890 (+ 426us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:54.634894 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:54.634896 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:54.635391 (+ 495us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:54.635762 (+ 371us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:54.636089 (+ 327us) server_negotiation.cc:299] Negotiation successful
0806 09:24:54.636373 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":644,"thread_start_us":448,"threads_started":1}
I20260806 09:24:54.638124 2795 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:24:54.735544 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 2611
I20260806 09:24:54.761291 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 2003
I20260806 09:24:54.763168 2099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:24:54.882141 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 2003
I20260806 09:24:54.918053 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:42405
--webserver_interface=127.16.86.190
--webserver_port=44889
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:42405
--ranger_config_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.86.148:58699/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.86.148:58699/kms with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:24:55.335654 2821 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:24:55.336180 2821 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:24:55.336376 2821 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:24:55.344348 2821 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:24:55.344517 2821 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:24:55.344636 2821 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:24:55.344754 2821 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:24:55.344867 2821 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:24:55.356148 2821 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.86.148:58699/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:42405
--ranger_config_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:42405
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=44889
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:24:55.357954 2821 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:24:55.359658 2821 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:24:55.374403 2828 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:55.374416 2826 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:55.375098 2831 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:24:55.380800 2821 server_base.cc:1034] running on GCE node
I20260806 09:24:55.382126 2821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:24:55.384375 2821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:24:55.385680 2821 hybrid_clock.cc:648] HybridClock initialized: now 1786008295385611 us; error 43 us; skew 500 ppm
I20260806 09:24:55.386197 2821 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:24:55 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008295, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:24:55.390229 2821 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:24:55.392584 2821 webserver.cc:460] Webserver started at http://127.16.86.190:44889/ using document root <none> and password file <none>
I20260806 09:24:55.393566 2821 fs_manager.cc:359] Metadata directory not provided
I20260806 09:24:55.393771 2821 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 09:24:55 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008295, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:24:55.450991 2821 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:24:55.455271 2845 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:55.457165 2821 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:24:55.457387 2821 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "1a0dd92539c940c68d966a15186817d4"
format_stamp: "Formatted at 2026-08-06 09:24:50 on dist-test-slave-h3ft"
server_key: "2214d484d54b30d1007897526ae3a156"
server_key_iv: "9234e21325bc6c3f9854ad34860b0f92"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:55.459215 2821 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:24:55.492437 2821 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:24:55.499626 2821 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:24:55.500028 2821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:24:55.527374 2821 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:42405
I20260806 09:24:55.527400 2910 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:42405 every 8 connection(s)
I20260806 09:24:55.529341 2821 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:24:55.537312 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 2821
I20260806 09:24:55.537695 2912 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:24:55.538470 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.86.148:58699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:44223
--local_ip_for_outbound_sockets=127.16.86.129
--tserver_master_addrs=127.16.86.190:42405
--webserver_port=38705
--webserver_interface=127.16.86.129
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.86.148:58699/kms with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:24:55.543133 2912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4: Bootstrap starting.
I20260806 09:24:55.565132 2912 log.cc:824] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4: Log is configured to *not* fsync() on all Append() calls
I20260806 09:24:55.575600 2912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:24:55.576215 2912 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4: Bootstrap complete.
I20260806 09:24:55.585990 2912 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0dd92539c940c68d966a15186817d4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42405 } }
I20260806 09:24:55.586935 2912 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a0dd92539c940c68d966a15186817d4, State: Initialized, Role: FOLLOWER
I20260806 09:24:55.587744 2912 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0dd92539c940c68d966a15186817d4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42405 } }
I20260806 09:24:55.587996 2912 raft_consensus.cc:388] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:24:55.588158 2912 raft_consensus.cc:482] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:24:55.588367 2912 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:24:55.592566 2912 raft_consensus.cc:504] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0dd92539c940c68d966a15186817d4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42405 } }
I20260806 09:24:55.593170 2912 leader_election.cc:302] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a0dd92539c940c68d966a15186817d4; no voters:
I20260806 09:24:55.593868 2912 leader_election.cc:288] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 09:24:55.594012 2919 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:24:55.595800 2919 raft_consensus.cc:686] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [term 2 LEADER]: Becoming Leader. State: Replica: 1a0dd92539c940c68d966a15186817d4, State: Running, Role: LEADER
I20260806 09:24:55.596422 2919 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0dd92539c940c68d966a15186817d4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42405 } }
I20260806 09:24:55.596691 2912 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:24:55.599459 2920 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "1a0dd92539c940c68d966a15186817d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0dd92539c940c68d966a15186817d4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42405 } } }
I20260806 09:24:55.599988 2920 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [sys.catalog]: This master's current role is: LEADER
I20260806 09:24:55.603909 2921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a0dd92539c940c68d966a15186817d4. Latest consensus state: current_term: 2 leader_uuid: "1a0dd92539c940c68d966a15186817d4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a0dd92539c940c68d966a15186817d4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 42405 } } }
I20260806 09:24:55.604261 2921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4 [sys.catalog]: This master's current role is: LEADER
I20260806 09:24:55.606161 2925 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:24:55.612696 2925 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:24:55.614257 2925 catalog_manager.cc:1259] Loaded cluster ID: 1d57ac99a1ef4a778e7a1988683628e3
I20260806 09:24:55.614370 2925 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:24:55.618770 2925 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:24:55.620462 2925 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4: Loaded TSK: 0
I20260806 09:24:55.622345 2925 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:24:55.626350 2912 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
W20260806 09:24:56.132232 2916 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:24:56.132845 2916 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:24:56.133134 2916 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:24:56.146471 2916 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:24:56.146677 2916 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:24:56.146924 2916 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:24:56.165360 2916 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.86.148:58699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:44223
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=38705
--tserver_master_addrs=127.16.86.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:24:56.167861 2916 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:24:56.170411 2916 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:24:56.201592 2959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:56.202457 2960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:56.211925 2962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:24:56.213267 2916 server_base.cc:1034] running on GCE node
I20260806 09:24:56.214051 2916 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:24:56.215329 2916 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:24:56.216589 2916 hybrid_clock.cc:648] HybridClock initialized: now 1786008296216528 us; error 63 us; skew 500 ppm
I20260806 09:24:56.217043 2916 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:24:56 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008296, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:24:56.220692 2916 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:24:56.222783 2916 webserver.cc:460] Webserver started at http://127.16.86.129:38705/ using document root <none> and password file <none>
I20260806 09:24:56.223662 2916 fs_manager.cc:359] Metadata directory not provided
I20260806 09:24:56.223837 2916 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 09:24:56 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008296, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:24:56.261865 2916 fs_manager.cc:709] Time spent opening directory manager: real 0.014s user 0.004s sys 0.007s
I20260806 09:24:56.266881 2976 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:56.269368 2916 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260806 09:24:56.269694 2916 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "b2d4de180c664567bc30ce07c92dfb8e"
format_stamp: "Formatted at 2026-08-06 09:24:52 on dist-test-slave-h3ft"
server_key: "97636af8fef18369845b44b2e6b518ee"
server_key_iv: "7e90a4da7619ed27ad4d33e3da290e10"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:56.271497 2916 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:24:56.329006 2916 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:24:56.335816 2916 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:24:56.336261 2916 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:24:56.337984 2916 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:24:56.340152 2916 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:24:56.340302 2916 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:56.340435 2916 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:24:56.340528 2916 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:56.533696 3141 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:44223 every 8 connection(s)
I20260806 09:24:56.534102 2916 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:44223
I20260806 09:24:56.536516 2916 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:24:56.542076 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 2916
I20260806 09:24:56.542894 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.86.148:58699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:37083
--local_ip_for_outbound_sockets=127.16.86.130
--tserver_master_addrs=127.16.86.190:42405
--webserver_port=34767
--webserver_interface=127.16.86.130
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.86.148:58699/kms with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:24:56 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008296, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:24:56.603227 3143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:56.546813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:35385 (local address 127.16.86.190:42405)
0806 09:24:56.567657 (+ 20844us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:56.567681 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:56.567746 (+ 65us) server_negotiation.cc:407] Connection header received
0806 09:24:56.568247 (+ 501us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:56.568271 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:56.568802 (+ 531us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:56.569306 (+ 504us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:56.578211 (+ 8905us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:56.580717 (+ 2506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:56.583765 (+ 3048us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:56.584914 (+ 1149us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:56.585097 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:56.592110 (+ 7013us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:56.592128 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:56.592156 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:56.592225 (+ 69us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:56.596372 (+ 4147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:56.597730 (+ 1358us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:56.597743 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:56.597752 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:56.598048 (+ 296us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:56.598770 (+ 722us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:56.598776 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:56.598780 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:56.599667 (+ 887us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:56.599884 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:56.601229 (+ 1345us) server_negotiation.cc:299] Negotiation successful
0806 09:24:56.601842 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":20501,"thread_start_us":23652,"threads_started":1}
I20260806 09:24:56.606880 3144 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:56.547174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:42405 (local address 127.16.86.129:35385)
0806 09:24:56.561341 (+ 14167us) negotiation.cc:107] Waiting for socket to connect
0806 09:24:56.561441 (+ 100us) client_negotiation.cc:174] Beginning negotiation
0806 09:24:56.564440 (+ 2999us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:24:56.575645 (+ 11205us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:24:56.575664 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:24:56.576600 (+ 936us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:24:56.577739 (+ 1139us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:56.577776 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:56.581160 (+ 3384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:56.581166 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:56.583516 (+ 2350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:56.583537 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:56.585657 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:56.585666 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:56.585975 (+ 309us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:56.587998 (+ 2023us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:24:56.588054 (+ 56us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:24:56.591569 (+ 3515us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:24:56.596687 (+ 5118us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:24:56.596701 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:24:56.596722 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:24:56.597508 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:24:56.598261 (+ 753us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:24:56.598279 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:24:56.598283 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:24:56.598611 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:24:56.600007 (+ 1396us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:24:56.600026 (+ 19us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:24:56.600562 (+ 536us) client_negotiation.cc:769] Sending connection context
0806 09:24:56.604159 (+ 3597us) client_negotiation.cc:240] Negotiation successful
0806 09:24:56.604873 (+ 714us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13706,"thread_start_us":17705,"threads_started":1}
I20260806 09:24:56.611752 3142 heartbeater.cc:347] Connected to a master server at 127.16.86.190:42405
I20260806 09:24:56.612226 3142 heartbeater.cc:464] Registering TS with master...
I20260806 09:24:56.613464 3142 heartbeater.cc:511] Master 127.16.86.190:42405 requested a full tablet report, sending...
W20260806 09:24:56.617722 3142 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:42405 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:24:56.618036 3142 heartbeater.cc:464] Registering TS with master...
I20260806 09:24:56.618417 3142 heartbeater.cc:511] Master 127.16.86.190:42405 requested a full tablet report, sending...
I20260806 09:24:56.619792 3142 heartbeater.cc:464] Registering TS with master...
I20260806 09:24:56.620081 3142 heartbeater.cc:511] Master 127.16.86.190:42405 requested a full tablet report, sending...
W20260806 09:24:56.621212 3142 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 09:24:56.743289 3157 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260806 09:24:57.111234 3147 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:24:57.111755 3147 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:24:57.111943 3147 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:24:57.125411 3147 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:24:57.125614 3147 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:24:57.125833 3147 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
Aug 06 09:24:57 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1786008297, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:24:57.144006 3147 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.86.148:58699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:37083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=34767
--tserver_master_addrs=127.16.86.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:24:57.146039 3147 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:24:57.148128 3147 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:24:57.169610 3169 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:57.170840 3170 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:57.187222 3173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:24:57.188040 3147 server_base.cc:1034] running on GCE node
I20260806 09:24:57.189023 3147 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:24:57.191898 3147 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:24:57.194672 3147 hybrid_clock.cc:648] HybridClock initialized: now 1786008297194565 us; error 99 us; skew 500 ppm
I20260806 09:24:57.195694 3147 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:24:57 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008297, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:24:57.205004 3147 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:24:57.207810 3147 webserver.cc:460] Webserver started at http://127.16.86.130:34767/ using document root <none> and password file <none>
I20260806 09:24:57.208984 3147 fs_manager.cc:359] Metadata directory not provided
I20260806 09:24:57.209187 3147 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 09:24:57 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008297, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:24:57.248525 3147 fs_manager.cc:709] Time spent opening directory manager: real 0.008s user 0.005s sys 0.000s
I20260806 09:24:57.253738 3189 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:57.256441 3147 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.005s sys 0.001s
I20260806 09:24:57.256734 3147 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "31548b00d49645b6baf237b840061801"
format_stamp: "Formatted at 2026-08-06 09:24:53 on dist-test-slave-h3ft"
server_key: "94a4c1b957b746931d44c92abb014c8e"
server_key_iv: "47a50bfaf868b8c8f88209dd4d5e2ba5"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:57.258575 3147 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:24:57.289896 3147 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:24:57.297029 3147 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:24:57.297488 3147 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:24:57.299343 3147 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:24:57.301718 3147 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:24:57.301872 3147 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:57.302017 3147 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:24:57.302114 3147 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:57.384979 3147 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:37083
I20260806 09:24:57.385128 3363 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:37083 every 8 connection(s)
I20260806 09:24:57.387483 3147 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:24:57.389285 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 3147
I20260806 09:24:57.390137 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.16.86.148:58699/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:39453
--local_ip_for_outbound_sockets=127.16.86.131
--tserver_master_addrs=127.16.86.190:42405
--webserver_port=46545
--webserver_interface=127.16.86.131
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:34621
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.16.86.148:58699/kms with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:24:57 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008297, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:24:57.456421 3369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:57.403431 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:54475 (local address 127.16.86.190:42405)
0806 09:24:57.433872 (+ 30441us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:57.433880 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:57.433914 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:24:57.434003 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:57.434009 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:57.434114 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:57.434249 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:57.437209 (+ 2960us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:57.438550 (+ 1341us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:57.441730 (+ 3180us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:57.442408 (+ 678us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:57.442517 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:57.448025 (+ 5508us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:57.448032 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:57.448038 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:57.448089 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:57.452151 (+ 4062us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:57.453269 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:57.453276 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:57.453280 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:57.453464 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:57.454143 (+ 679us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:57.454148 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:57.454152 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:57.454493 (+ 341us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:57.454684 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:57.455655 (+ 971us) server_negotiation.cc:299] Negotiation successful
0806 09:24:57.455964 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30250,"thread_start_us":30951,"threads_started":1}
I20260806 09:24:57.457646 3370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:57.403760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:42405 (local address 127.16.86.130:54475)
0806 09:24:57.429644 (+ 25884us) negotiation.cc:107] Waiting for socket to connect
0806 09:24:57.429729 (+ 85us) client_negotiation.cc:174] Beginning negotiation
0806 09:24:57.432692 (+ 2963us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:24:57.434956 (+ 2264us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:24:57.434973 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:24:57.435839 (+ 866us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:24:57.436883 (+ 1044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:57.436916 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:57.438822 (+ 1906us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:57.438828 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:57.441527 (+ 2699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:57.441544 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:57.442977 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:57.442986 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:57.443378 (+ 392us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:57.444386 (+ 1008us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:24:57.444434 (+ 48us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:24:57.447678 (+ 3244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:24:57.452367 (+ 4689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:24:57.452377 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:24:57.452410 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:24:57.453108 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:24:57.453630 (+ 522us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:24:57.453634 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:24:57.453637 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:24:57.453981 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:24:57.454643 (+ 662us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:24:57.454652 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:24:57.455234 (+ 582us) client_negotiation.cc:769] Sending connection context
0806 09:24:57.455552 (+ 318us) client_negotiation.cc:240] Negotiation successful
0806 09:24:57.456261 (+ 709us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":23603,"thread_start_us":24225,"threads_started":1}
I20260806 09:24:57.459533 3364 heartbeater.cc:347] Connected to a master server at 127.16.86.190:42405
I20260806 09:24:57.459941 3364 heartbeater.cc:464] Registering TS with master...
I20260806 09:24:57.461113 3364 heartbeater.cc:511] Master 127.16.86.190:42405 requested a full tablet report, sending...
I20260806 09:24:57.463766 2869 ts_manager.cc:194] Registered new tserver with Master: 31548b00d49645b6baf237b840061801 (127.16.86.130:37083)
I20260806 09:24:57.466078 2869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:54475
I20260806 09:24:57.621464 3142 heartbeater.cc:464] Registering TS with master...
I20260806 09:24:57.622068 3142 heartbeater.cc:511] Master 127.16.86.190:42405 requested a full tablet report, sending...
I20260806 09:24:57.623742 2869 ts_manager.cc:194] Registered new tserver with Master: b2d4de180c664567bc30ce07c92dfb8e (127.16.86.129:44223)
I20260806 09:24:57.625111 2869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:35385
Aug 06 09:24:57 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008297, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
W20260806 09:24:57.886520 3368 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:24:57.886895 3368 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:24:57.887131 3368 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:24:57.895785 3368 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:24:57.895946 3368 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:24:57.896124 3368 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:24:57.911777 3368 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:34621
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.16.86.148:58699/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:39453
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=46545
--tserver_master_addrs=127.16.86.190:42405
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:24:57.913697 3368 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:24:57.915540 3368 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:24:57.934613 3379 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:24:57.935176 3381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:24:57.941232 3368 server_base.cc:1034] running on GCE node
W20260806 09:24:57.941596 3384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:24:57.942739 3368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:24:57.944526 3368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:24:57.945922 3368 hybrid_clock.cc:648] HybridClock initialized: now 1786008297945761 us; error 147 us; skew 500 ppm
I20260806 09:24:57.946426 3368 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:24:57 dist-test-slave-h3ft krb5kdc[1361](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008297, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:24:57.952340 3368 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:24:57.955148 3368 webserver.cc:460] Webserver started at http://127.16.86.131:46545/ using document root <none> and password file <none>
I20260806 09:24:57.956269 3368 fs_manager.cc:359] Metadata directory not provided
I20260806 09:24:57.956462 3368 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
Aug 06 09:24:57 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008297, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:24:57.985652 3368 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.003s
I20260806 09:24:57.990321 3392 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:57.992941 3368 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 09:24:57.993295 3368 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "88b022611a4f4cb5a5fead1585ebfc88"
format_stamp: "Formatted at 2026-08-06 09:24:54 on dist-test-slave-h3ft"
server_key: "b28ef32c54e36210af4158ad66aaaff7"
server_key_iv: "d224ab15ce4d4edd714797a5975efb12"
server_key_version: "kuduclusterkey@0"
I20260806 09:24:57.995231 3368 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:24:58.041656 3368 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:24:58.048964 3368 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:24:58.049445 3368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:24:58.063652 3368 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
Aug 06 09:24:58 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008297, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for HTTP/127.16.86.148@KRBTEST.COM
I20260806 09:24:58.078425 3368 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:24:58.078722 3368 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:58.078886 3368 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:24:58.078984 3368 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:24:58.241547 3368 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:39453
I20260806 09:24:58.241561 3567 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:39453 every 8 connection(s)
I20260806 09:24:58.243623 3368 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:24:58.251931 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 3368
Aug 06 09:24:58 dist-test-slave-h3ft krb5kdc[1361](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008297, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:24:58.316205 3572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.258729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:58793 (local address 127.16.86.190:42405)
0806 09:24:58.291542 (+ 32813us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:58.291549 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:58.291574 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:24:58.291644 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:58.291649 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:58.292409 (+ 760us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:58.292580 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:58.296459 (+ 3879us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.297869 (+ 1410us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.300614 (+ 2745us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.301378 (+ 764us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.301491 (+ 113us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.307168 (+ 5677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:58.307176 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:58.307181 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:58.307241 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:58.311420 (+ 4179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:58.312612 (+ 1192us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:58.312620 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:58.312625 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:58.312869 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:58.313572 (+ 703us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:58.313578 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:58.313581 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:58.314040 (+ 459us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:58.314196 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:58.315474 (+ 1278us) server_negotiation.cc:299] Negotiation successful
0806 09:24:58.315783 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32648,"thread_start_us":26229,"threads_started":1}
I20260806 09:24:58.319180 3574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.257867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:42405 (local address 127.16.86.131:58793)
0806 09:24:58.286616 (+ 28749us) negotiation.cc:107] Waiting for socket to connect
0806 09:24:58.286715 (+ 99us) client_negotiation.cc:174] Beginning negotiation
0806 09:24:58.289796 (+ 3081us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:24:58.294237 (+ 4441us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:24:58.294253 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:24:58.295074 (+ 821us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:24:58.296187 (+ 1113us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:58.296215 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.298139 (+ 1924us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.298145 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:58.300428 (+ 2283us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:58.300444 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.301942 (+ 1498us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.301948 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:58.302262 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.303422 (+ 1160us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:24:58.303459 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:24:58.306748 (+ 3289us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:24:58.311628 (+ 4880us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:24:58.311640 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:24:58.311666 (+ 26us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:24:58.312399 (+ 733us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:24:58.313045 (+ 646us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:24:58.313050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:24:58.313053 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:24:58.313386 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:24:58.314306 (+ 920us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:24:58.314317 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:24:58.314908 (+ 591us) client_negotiation.cc:769] Sending connection context
0806 09:24:58.316617 (+ 1709us) client_negotiation.cc:240] Negotiation successful
0806 09:24:58.317360 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":28231,"thread_start_us":32383,"threads_started":1}
I20260806 09:24:58.320689 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.253936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60978 (local address 127.16.86.190:42405)
0806 09:24:58.284332 (+ 30396us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:58.284340 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:58.284374 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:24:58.284473 (+ 99us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:58.284481 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:58.285602 (+ 1121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:58.285810 (+ 208us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:58.295473 (+ 9663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.296801 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.301049 (+ 4248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.302053 (+ 1004us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.302216 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.311740 (+ 9524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:58.311749 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:58.311755 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:58.311826 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:58.315976 (+ 4150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:58.317067 (+ 1091us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:58.317073 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:58.317077 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:58.317262 (+ 185us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:58.317851 (+ 589us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:58.317857 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:58.317861 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:58.318265 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:58.318571 (+ 306us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:58.319968 (+ 1397us) server_negotiation.cc:299] Negotiation successful
0806 09:24:58.320246 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30192,"thread_start_us":38937,"threads_started":1}
I20260806 09:24:58.321298 3569 heartbeater.cc:347] Connected to a master server at 127.16.86.190:42405
I20260806 09:24:58.321724 3569 heartbeater.cc:464] Registering TS with master...
I20260806 09:24:58.322855 3569 heartbeater.cc:511] Master 127.16.86.190:42405 requested a full tablet report, sending...
I20260806 09:24:58.324904 2869 ts_manager.cc:194] Registered new tserver with Master: 88b022611a4f4cb5a5fead1585ebfc88 (127.16.86.131:39453)
I20260806 09:24:58.326229 2869 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:58793
I20260806 09:24:58.337611 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:24:58.375787 3575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.343821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60984 (local address 127.16.86.190:42405)
0806 09:24:58.344074 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:58.344084 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:58.355899 (+ 11815us) server_negotiation.cc:407] Connection header received
0806 09:24:58.356399 (+ 500us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:58.356406 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:58.356509 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:58.356599 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:58.359162 (+ 2563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.360423 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.362557 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.363375 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.363601 (+ 226us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.367203 (+ 3602us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:58.367214 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:58.367222 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:58.367295 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:58.371493 (+ 4198us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:58.372593 (+ 1100us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:58.372602 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:58.372607 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:58.372838 (+ 231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:58.373468 (+ 630us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:58.373474 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:58.373479 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:58.373901 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:58.374286 (+ 385us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:58.375048 (+ 762us) server_negotiation.cc:299] Negotiation successful
0806 09:24:58.375358 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 09:24:58.383671 2869 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60984:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:24:58.396911 2869 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:24:58.440066 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.424178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49138 (local address 127.16.86.131:39453)
0806 09:24:58.425616 (+ 1438us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:58.425630 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:58.427981 (+ 2351us) server_negotiation.cc:407] Connection header received
0806 09:24:58.429282 (+ 1301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:58.429290 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:58.429533 (+ 243us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:58.429728 (+ 195us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:24:58.431122 (+ 1394us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.432916 (+ 1794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.436022 (+ 3106us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.437804 (+ 1782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.437974 (+ 170us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.438278 (+ 304us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:58.439357 (+ 1079us) server_negotiation.cc:299] Negotiation successful
0806 09:24:58.439632 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1207,"thread_start_us":1008,"threads_started":1}
I20260806 09:24:58.441030 3605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.424745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:39453 (local address 127.0.0.1:49138)
0806 09:24:58.427732 (+ 2987us) negotiation.cc:107] Waiting for socket to connect
0806 09:24:58.427758 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:24:58.429034 (+ 1276us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:24:58.430085 (+ 1051us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:24:58.430094 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:24:58.430137 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:24:58.430792 (+ 655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:58.430815 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.433128 (+ 2313us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.433134 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:58.435778 (+ 2644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:58.435797 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.438390 (+ 2593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.438402 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:58.438671 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.438750 (+ 79us) client_negotiation.cc:769] Sending connection context
0806 09:24:58.438916 (+ 166us) client_negotiation.cc:240] Negotiation successful
0806 09:24:58.440455 (+ 1539us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2846,"mutex_wait_us":29,"thread_start_us":1449,"threads_started":1}
I20260806 09:24:58.442102 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.423584 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52178 (local address 127.16.86.130:37083)
0806 09:24:58.426275 (+ 2691us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:58.426285 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:58.428071 (+ 1786us) server_negotiation.cc:407] Connection header received
0806 09:24:58.428211 (+ 140us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:58.428219 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:58.428542 (+ 323us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:58.428755 (+ 213us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:24:58.431387 (+ 2632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.434782 (+ 3395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.437944 (+ 3162us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.439637 (+ 1693us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.439775 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.440032 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:58.441449 (+ 1417us) server_negotiation.cc:299] Negotiation successful
0806 09:24:58.441676 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2457,"thread_start_us":2247,"threads_started":1}
I20260806 09:24:58.443423 3603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.425462 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:44223 (local address 127.0.0.1:51818)
0806 09:24:58.430927 (+ 5465us) negotiation.cc:107] Waiting for socket to connect
0806 09:24:58.430959 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 09:24:58.431143 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:24:58.432331 (+ 1188us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:24:58.432338 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:24:58.432365 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:24:58.432910 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:58.432923 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.435036 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.435042 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:58.440051 (+ 5009us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:58.440070 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.442528 (+ 2458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.442533 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:58.442673 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.442697 (+ 24us) client_negotiation.cc:769] Sending connection context
0806 09:24:58.442830 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 09:24:58.442991 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5297,"mutex_wait_us":2485,"thread_start_us":3971,"threads_started":1}
I20260806 09:24:58.443797 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.423686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51818 (local address 127.16.86.129:44223)
0806 09:24:58.429954 (+ 6268us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:58.429969 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:58.431288 (+ 1319us) server_negotiation.cc:407] Connection header received
0806 09:24:58.431426 (+ 138us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:58.431434 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:58.431764 (+ 330us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:58.432026 (+ 262us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:24:58.433157 (+ 1131us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.434822 (+ 1665us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.440294 (+ 5472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.441938 (+ 1644us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.442096 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.442374 (+ 278us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:58.442886 (+ 512us) server_negotiation.cc:299] Negotiation successful
0806 09:24:58.443118 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":588,"threads_started":1}
I20260806 09:24:58.441231 3604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.424745 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:37083 (local address 127.0.0.1:52178)
0806 09:24:58.426765 (+ 2020us) negotiation.cc:107] Waiting for socket to connect
0806 09:24:58.426828 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 09:24:58.427113 (+ 285us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:24:58.430471 (+ 3358us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:24:58.430479 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:24:58.430520 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:24:58.431107 (+ 587us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:58.431122 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.435037 (+ 3915us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.435042 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:58.437691 (+ 2649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:58.437712 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.440215 (+ 2503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.440221 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:58.440414 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.440452 (+ 38us) client_negotiation.cc:769] Sending connection context
0806 09:24:58.440539 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 09:24:58.440709 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1673,"mutex_wait_us":133,"thread_start_us":5549,"threads_started":1}
I20260806 09:24:58.448613 3052 tablet_service.cc:1467] Processing CreateTablet for tablet efdb03972ac24f1d999bbb8d90e37a7c (DEFAULT_TABLE table=test-table [id=06b326aa9c8c49ab94aedf6a28875716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:24:58.448657 3275 tablet_service.cc:1467] Processing CreateTablet for tablet efdb03972ac24f1d999bbb8d90e37a7c (DEFAULT_TABLE table=test-table [id=06b326aa9c8c49ab94aedf6a28875716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:24:58.448657 3468 tablet_service.cc:1467] Processing CreateTablet for tablet efdb03972ac24f1d999bbb8d90e37a7c (DEFAULT_TABLE table=test-table [id=06b326aa9c8c49ab94aedf6a28875716]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:24:58.450510 3052 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efdb03972ac24f1d999bbb8d90e37a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:24:58.450572 3468 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efdb03972ac24f1d999bbb8d90e37a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:24:58.452045 3275 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet efdb03972ac24f1d999bbb8d90e37a7c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:24:58.471850 3364 heartbeater.cc:503] Master 127.16.86.190:42405 was elected leader, sending a full tablet report...
I20260806 09:24:58.475241 3609 tablet_bootstrap.cc:492] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88: Bootstrap starting.
I20260806 09:24:58.480495 3607 tablet_bootstrap.cc:492] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e: Bootstrap starting.
I20260806 09:24:58.482524 3609 tablet_bootstrap.cc:654] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88: Neither blocks nor log segments found. Creating new log.
I20260806 09:24:58.484171 3609 log.cc:824] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88: Log is configured to *not* fsync() on all Append() calls
I20260806 09:24:58.484252 3611 tablet_bootstrap.cc:492] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801: Bootstrap starting.
I20260806 09:24:58.484709 3607 tablet_bootstrap.cc:654] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e: Neither blocks nor log segments found. Creating new log.
I20260806 09:24:58.486361 3607 log.cc:824] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:24:58.490492 3611 tablet_bootstrap.cc:654] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801: Neither blocks nor log segments found. Creating new log.
I20260806 09:24:58.491994 3611 log.cc:824] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801: Log is configured to *not* fsync() on all Append() calls
I20260806 09:24:58.497185 3609 tablet_bootstrap.cc:492] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88: No bootstrap required, opened a new log
I20260806 09:24:58.497637 3609 ts_tablet_manager.cc:1397] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88: Time spent bootstrapping tablet: real 0.023s user 0.012s sys 0.005s
I20260806 09:24:58.501847 3611 tablet_bootstrap.cc:492] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801: No bootstrap required, opened a new log
I20260806 09:24:58.502246 3611 ts_tablet_manager.cc:1397] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.009s
I20260806 09:24:58.513681 3607 tablet_bootstrap.cc:492] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e: No bootstrap required, opened a new log
I20260806 09:24:58.513535 3611 raft_consensus.cc:348] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31548b00d49645b6baf237b840061801" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 37083 } } peers { permanent_uuid: "b2d4de180c664567bc30ce07c92dfb8e" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 44223 } } peers { permanent_uuid: "88b022611a4f4cb5a5fead1585ebfc88" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 39453 } }
I20260806 09:24:58.514083 3611 raft_consensus.cc:374] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:24:58.514112 3607 ts_tablet_manager.cc:1397] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e: Time spent bootstrapping tablet: real 0.034s user 0.021s sys 0.009s
I20260806 09:24:58.514199 3611 raft_consensus.cc:729] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 31548b00d49645b6baf237b840061801, State: Initialized, Role: FOLLOWER
I20260806 09:24:58.515048 3611 consensus_queue.cc:260] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31548b00d49645b6baf237b840061801" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 37083 } } peers { permanent_uuid: "b2d4de180c664567bc30ce07c92dfb8e" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 44223 } } peers { permanent_uuid: "88b022611a4f4cb5a5fead1585ebfc88" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 39453 } }
I20260806 09:24:58.515692 3609 raft_consensus.cc:348] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31548b00d49645b6baf237b840061801" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 37083 } } peers { permanent_uuid: "b2d4de180c664567bc30ce07c92dfb8e" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 44223 } } peers { permanent_uuid: "88b022611a4f4cb5a5fead1585ebfc88" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 39453 } }
I20260806 09:24:58.516208 3609 raft_consensus.cc:374] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:24:58.516319 3609 raft_consensus.cc:729] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88b022611a4f4cb5a5fead1585ebfc88, State: Initialized, Role: FOLLOWER
I20260806 09:24:58.517189 3609 consensus_queue.cc:260] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31548b00d49645b6baf237b840061801" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 37083 } } peers { permanent_uuid: "b2d4de180c664567bc30ce07c92dfb8e" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 44223 } } peers { permanent_uuid: "88b022611a4f4cb5a5fead1585ebfc88" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 39453 } }
I20260806 09:24:58.523468 3611 ts_tablet_manager.cc:1428] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801: Time spent starting tablet: real 0.021s user 0.013s sys 0.007s
I20260806 09:24:58.526608 3609 ts_tablet_manager.cc:1428] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88: Time spent starting tablet: real 0.029s user 0.013s sys 0.011s
I20260806 09:24:58.528301 3607 raft_consensus.cc:348] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31548b00d49645b6baf237b840061801" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 37083 } } peers { permanent_uuid: "b2d4de180c664567bc30ce07c92dfb8e" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 44223 } } peers { permanent_uuid: "88b022611a4f4cb5a5fead1585ebfc88" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 39453 } }
I20260806 09:24:58.528854 3607 raft_consensus.cc:374] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:24:58.528972 3607 raft_consensus.cc:729] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b2d4de180c664567bc30ce07c92dfb8e, State: Initialized, Role: FOLLOWER
I20260806 09:24:58.529711 3607 consensus_queue.cc:260] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31548b00d49645b6baf237b840061801" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 37083 } } peers { permanent_uuid: "b2d4de180c664567bc30ce07c92dfb8e" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 44223 } } peers { permanent_uuid: "88b022611a4f4cb5a5fead1585ebfc88" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 39453 } }
I20260806 09:24:58.539451 3569 heartbeater.cc:503] Master 127.16.86.190:42405 was elected leader, sending a full tablet report...
W20260806 09:24:58.544129 3570 tablet.cc:2369] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:24:58.551121 3142 heartbeater.cc:503] Master 127.16.86.190:42405 was elected leader, sending a full tablet report...
I20260806 09:24:58.552083 3607 ts_tablet_manager.cc:1428] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e: Time spent starting tablet: real 0.038s user 0.025s sys 0.014s
W20260806 09:24:58.562275 3146 tablet.cc:2369] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:24:58.652861 3367 tablet.cc:2369] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:24:58.712141 3620 raft_consensus.cc:482] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:24:58.712577 3620 raft_consensus.cc:504] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31548b00d49645b6baf237b840061801" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 37083 } } peers { permanent_uuid: "b2d4de180c664567bc30ce07c92dfb8e" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 44223 } } peers { permanent_uuid: "88b022611a4f4cb5a5fead1585ebfc88" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 39453 } }
I20260806 09:24:58.714938 3620 leader_election.cc:288] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 31548b00d49645b6baf237b840061801 (127.16.86.130:37083), 88b022611a4f4cb5a5fead1585ebfc88 (127.16.86.131:39453)
I20260806 09:24:58.726257 3602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.715686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:58477 (local address 127.16.86.131:39453)
0806 09:24:58.715989 (+ 303us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:58.715999 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:58.718063 (+ 2064us) server_negotiation.cc:407] Connection header received
0806 09:24:58.718310 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:58.718316 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:58.718396 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:58.718506 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:24:58.719473 (+ 967us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.720279 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.723632 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.724945 (+ 1313us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.725156 (+ 211us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.725243 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:58.725691 (+ 448us) server_negotiation.cc:299] Negotiation successful
0806 09:24:58.725890 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 09:24:58.726753 3621 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.715762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:39453 (local address 127.16.86.129:58477)
0806 09:24:58.717903 (+ 2141us) negotiation.cc:107] Waiting for socket to connect
0806 09:24:58.717932 (+ 29us) client_negotiation.cc:174] Beginning negotiation
0806 09:24:58.718132 (+ 200us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:24:58.718666 (+ 534us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:24:58.718669 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:24:58.718692 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:24:58.719283 (+ 591us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:58.719299 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.720433 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.720441 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:58.723379 (+ 2938us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:58.723396 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.725258 (+ 1862us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.725264 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:58.725433 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.725474 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:24:58.725647 (+ 173us) client_negotiation.cc:240] Negotiation successful
0806 09:24:58.725890 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1968,"mutex_wait_us":215,"thread_start_us":1191,"threads_started":1}
I20260806 09:24:58.728107 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.716339 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:39213 (local address 127.16.86.130:37083)
0806 09:24:58.716623 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:58.716631 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:58.717657 (+ 1026us) server_negotiation.cc:407] Connection header received
0806 09:24:58.718056 (+ 399us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:58.718061 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:58.718166 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:58.718283 (+ 117us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:24:58.719221 (+ 938us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.720437 (+ 1216us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.725349 (+ 4912us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.726622 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.726812 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.726891 (+ 79us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:58.727653 (+ 762us) server_negotiation.cc:299] Negotiation successful
0806 09:24:58.727810 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 09:24:58.729017 3622 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.715476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:37083 (local address 127.16.86.129:39213)
0806 09:24:58.717495 (+ 2019us) negotiation.cc:107] Waiting for socket to connect
0806 09:24:58.717522 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:24:58.717828 (+ 306us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:24:58.718342 (+ 514us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:24:58.718347 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:24:58.718373 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:24:58.718933 (+ 560us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:58.718949 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.720581 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.720586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:58.723379 (+ 2793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:24:58.723395 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.727121 (+ 3726us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.727132 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:24:58.727390 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.727460 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 09:24:58.727592 (+ 132us) client_negotiation.cc:240] Negotiation successful
0806 09:24:58.727763 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1798,"mutex_wait_us":117,"thread_start_us":1998,"threads_started":1}
I20260806 09:24:58.728771 3495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efdb03972ac24f1d999bbb8d90e37a7c" candidate_uuid: "b2d4de180c664567bc30ce07c92dfb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b022611a4f4cb5a5fead1585ebfc88" is_pre_election: true
I20260806 09:24:58.730083 3495 raft_consensus.cc:2454] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2d4de180c664567bc30ce07c92dfb8e in term 0.
I20260806 09:24:58.731298 2984 leader_election.cc:302] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b022611a4f4cb5a5fead1585ebfc88, b2d4de180c664567bc30ce07c92dfb8e; no voters:
I20260806 09:24:58.731567 3302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efdb03972ac24f1d999bbb8d90e37a7c" candidate_uuid: "b2d4de180c664567bc30ce07c92dfb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31548b00d49645b6baf237b840061801" is_pre_election: true
I20260806 09:24:58.731961 3620 raft_consensus.cc:2781] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:24:58.732218 3620 raft_consensus.cc:482] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:24:58.732214 3302 raft_consensus.cc:2454] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b2d4de180c664567bc30ce07c92dfb8e in term 0.
I20260806 09:24:58.732430 3620 raft_consensus.cc:3037] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:24:58.739668 3620 raft_consensus.cc:504] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31548b00d49645b6baf237b840061801" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 37083 } } peers { permanent_uuid: "b2d4de180c664567bc30ce07c92dfb8e" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 44223 } } peers { permanent_uuid: "88b022611a4f4cb5a5fead1585ebfc88" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 39453 } }
I20260806 09:24:58.740665 3620 leader_election.cc:288] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [CANDIDATE]: Term 1 election: Requested vote from peers 31548b00d49645b6baf237b840061801 (127.16.86.130:37083), 88b022611a4f4cb5a5fead1585ebfc88 (127.16.86.131:39453)
I20260806 09:24:58.741303 3302 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efdb03972ac24f1d999bbb8d90e37a7c" candidate_uuid: "b2d4de180c664567bc30ce07c92dfb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "31548b00d49645b6baf237b840061801"
I20260806 09:24:58.741595 3302 raft_consensus.cc:3037] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:24:58.741568 3495 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "efdb03972ac24f1d999bbb8d90e37a7c" candidate_uuid: "b2d4de180c664567bc30ce07c92dfb8e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "88b022611a4f4cb5a5fead1585ebfc88"
I20260806 09:24:58.741873 3495 raft_consensus.cc:3037] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:24:58.745087 3495 raft_consensus.cc:2454] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2d4de180c664567bc30ce07c92dfb8e in term 1.
I20260806 09:24:58.745486 3302 raft_consensus.cc:2454] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b2d4de180c664567bc30ce07c92dfb8e in term 1.
I20260806 09:24:58.745836 2984 leader_election.cc:302] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 88b022611a4f4cb5a5fead1585ebfc88, b2d4de180c664567bc30ce07c92dfb8e; no voters:
I20260806 09:24:58.746299 3620 raft_consensus.cc:2781] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:24:58.747438 3620 raft_consensus.cc:686] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 1 LEADER]: Becoming Leader. State: Replica: b2d4de180c664567bc30ce07c92dfb8e, State: Running, Role: LEADER
I20260806 09:24:58.748287 3620 consensus_queue.cc:237] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31548b00d49645b6baf237b840061801" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 37083 } } peers { permanent_uuid: "b2d4de180c664567bc30ce07c92dfb8e" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 44223 } } peers { permanent_uuid: "88b022611a4f4cb5a5fead1585ebfc88" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 39453 } }
I20260806 09:24:58.756093 2867 catalog_manager.cc:5495] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e reported cstate change: term changed from 0 to 1, leader changed from <none> to b2d4de180c664567bc30ce07c92dfb8e (127.16.86.129). New cstate: current_term: 1 leader_uuid: "b2d4de180c664567bc30ce07c92dfb8e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "31548b00d49645b6baf237b840061801" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 37083 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b2d4de180c664567bc30ce07c92dfb8e" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 44223 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "88b022611a4f4cb5a5fead1585ebfc88" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 39453 } health_report { overall_health: UNKNOWN } } }
I20260806 09:24:58.818538 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.811567 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59758 (local address 127.16.86.129:44223)
0806 09:24:58.811909 (+ 342us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:58.811915 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:58.811986 (+ 71us) server_negotiation.cc:407] Connection header received
0806 09:24:58.812229 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:58.812234 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:58.812332 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:58.812472 (+ 140us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:24:58.813113 (+ 641us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.813889 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.816037 (+ 2148us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.816440 (+ 403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.816544 (+ 104us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.817352 (+ 808us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:24:58.817899 (+ 547us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:24:58.818013 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:58.818142 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 09:24:58.818216 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":112}
I20260806 09:24:58.829420 3495 raft_consensus.cc:1261] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88 [term 1 FOLLOWER]: Refusing update from remote peer b2d4de180c664567bc30ce07c92dfb8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:24:58.829802 3302 raft_consensus.cc:1261] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801 [term 1 FOLLOWER]: Refusing update from remote peer b2d4de180c664567bc30ce07c92dfb8e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:24:58.830552 3624 consensus_queue.cc:1035] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "88b022611a4f4cb5a5fead1585ebfc88" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 39453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:24:58.831122 3620 consensus_queue.cc:1035] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [LEADER]: Connected to new peer: Peer: permanent_uuid: "31548b00d49645b6baf237b840061801" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 37083 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:24:58.869081 3628 mvcc.cc:205] Tried to move back new op lower bound from 7315489991991435264 to 7315489991678742528. Current Snapshot: MvccSnapshot[applied={T|T < 7315489991991435264}]
I20260806 09:24:58.878229 2867 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:24:58.878841 2867 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:60984:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:24:58.883260 2867 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 1a0dd92539c940c68d966a15186817d4: Sending DeleteTablet for 3 replicas of tablet efdb03972ac24f1d999bbb8d90e37a7c
I20260806 09:24:58.884758 3468 tablet_service.cc:1514] Processing DeleteTablet for tablet efdb03972ac24f1d999bbb8d90e37a7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:24:58 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:49138
I20260806 09:24:58.884809 3052 tablet_service.cc:1514] Processing DeleteTablet for tablet efdb03972ac24f1d999bbb8d90e37a7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:24:58 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:51818
I20260806 09:24:58.885119 3275 tablet_service.cc:1514] Processing DeleteTablet for tablet efdb03972ac24f1d999bbb8d90e37a7c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:24:58 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:52178
I20260806 09:24:58.892514 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 2916
I20260806 09:24:58.900166 3641 tablet_replica.cc:330] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e: stopping tablet replica
I20260806 09:24:58.900790 3641 raft_consensus.cc:2229] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:24:58.901679 3641 raft_consensus.cc:2258] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:24:58.903781 3639 tablet_replica.cc:330] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88: stopping tablet replica
I20260806 09:24:58.904280 3639 raft_consensus.cc:2229] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:24:58.904395 3642 tablet_replica.cc:330] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801: stopping tablet replica
I20260806 09:24:58.905153 3639 raft_consensus.cc:2258] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:24:58.905220 3641 ts_tablet_manager.cc:1905] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:24:58.905351 3642 raft_consensus.cc:2229] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:24:58.906494 3642 raft_consensus.cc:2258] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:24:58.908993 3642 ts_tablet_manager.cc:1905] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:24:58.917500 3641 ts_tablet_manager.cc:1918] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:24:58.917975 3641 log.cc:1196] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/efdb03972ac24f1d999bbb8d90e37a7c
I20260806 09:24:58.919173 3641 ts_tablet_manager.cc:1939] T efdb03972ac24f1d999bbb8d90e37a7c P b2d4de180c664567bc30ce07c92dfb8e: Deleting consensus metadata
I20260806 09:24:58.920133 3639 ts_tablet_manager.cc:1905] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:24:58.922019 2848 catalog_manager.cc:4841] TS b2d4de180c664567bc30ce07c92dfb8e (127.16.86.129:44223): tablet efdb03972ac24f1d999bbb8d90e37a7c (table test-table [id=06b326aa9c8c49ab94aedf6a28875716]) successfully deleted
I20260806 09:24:58.923447 3642 ts_tablet_manager.cc:1918] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:24:58.923822 3642 log.cc:1196] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/efdb03972ac24f1d999bbb8d90e37a7c
I20260806 09:24:58.924695 3642 ts_tablet_manager.cc:1939] T efdb03972ac24f1d999bbb8d90e37a7c P 31548b00d49645b6baf237b840061801: Deleting consensus metadata
I20260806 09:24:58.927357 2846 catalog_manager.cc:4841] TS 31548b00d49645b6baf237b840061801 (127.16.86.130:37083): tablet efdb03972ac24f1d999bbb8d90e37a7c (table test-table [id=06b326aa9c8c49ab94aedf6a28875716]) successfully deleted
I20260806 09:24:58.930917 3639 ts_tablet_manager.cc:1918] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:24:58.931314 3639 log.cc:1196] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/efdb03972ac24f1d999bbb8d90e37a7c
I20260806 09:24:58.932356 3639 ts_tablet_manager.cc:1939] T efdb03972ac24f1d999bbb8d90e37a7c P 88b022611a4f4cb5a5fead1585ebfc88: Deleting consensus metadata
I20260806 09:24:58.934391 2849 catalog_manager.cc:4841] TS 88b022611a4f4cb5a5fead1585ebfc88 (127.16.86.131:39453): tablet efdb03972ac24f1d999bbb8d90e37a7c (table test-table [id=06b326aa9c8c49ab94aedf6a28875716]) successfully deleted
I20260806 09:24:58.937197 3600 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:58.894385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59760 (local address 127.16.86.129:44223)
0806 09:24:58.894678 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:58.894689 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:58.910516 (+ 15827us) server_negotiation.cc:407] Connection header received
0806 09:24:58.910809 (+ 293us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:58.910815 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:58.910982 (+ 167us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:58.911138 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:58.913584 (+ 2446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.915371 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.918057 (+ 2686us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:58.918948 (+ 891us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:58.919093 (+ 145us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:58.924726 (+ 5633us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:58.924746 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:58.924776 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:58.924840 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:58.932336 (+ 7496us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:58.933865 (+ 1529us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:58.933876 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:58.933880 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:58.934249 (+ 369us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:58.934975 (+ 726us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:58.934979 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:58.934982 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:58.935584 (+ 602us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:58.935899 (+ 315us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:58.936423 (+ 524us) server_negotiation.cc:299] Negotiation successful
0806 09:24:58.936654 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 09:24:58.938333 3137 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:24:59.044643 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 2916
I20260806 09:24:59.077663 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 3147
I20260806 09:24:59.095295 3599 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:59.078860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60652 (local address 127.16.86.130:37083)
0806 09:24:59.079140 (+ 280us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:59.079145 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:59.079192 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:24:59.079516 (+ 324us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:59.079521 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:59.079602 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:59.079712 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:59.081610 (+ 1898us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:59.082354 (+ 744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:59.084010 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:59.084463 (+ 453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:59.084585 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:59.088030 (+ 3445us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:59.088047 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:59.088064 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:59.088114 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:59.091968 (+ 3854us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:59.093032 (+ 1064us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:59.093040 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:59.093044 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:59.093233 (+ 189us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:59.093665 (+ 432us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:59.093669 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:59.093671 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:59.094068 (+ 397us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:59.094335 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:59.094705 (+ 370us) server_negotiation.cc:299] Negotiation successful
0806 09:24:59.094896 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:24:59.096011 3359 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:24:59.197252 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 3147
I20260806 09:24:59.227995 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 3368
I20260806 09:24:59.245710 3648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:24:59.229106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33960 (local address 127.16.86.131:39453)
0806 09:24:59.229804 (+ 698us) server_negotiation.cc:206] Beginning negotiation
0806 09:24:59.229809 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:24:59.229834 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:24:59.229902 (+ 68us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:24:59.229906 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:24:59.229980 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:24:59.230148 (+ 168us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:24:59.232049 (+ 1901us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:59.232863 (+ 814us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:59.234297 (+ 1434us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:24:59.234713 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:24:59.234818 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:24:59.238128 (+ 3310us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:24:59.238146 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:24:59.238163 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:24:59.238200 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:24:59.242242 (+ 4042us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:59.243222 (+ 980us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:59.243231 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:59.243235 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:59.243427 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:24:59.243901 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:24:59.243905 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:24:59.243909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:24:59.244393 (+ 484us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:24:59.244707 (+ 314us) server_negotiation.cc:1035] Waiting for connection context
0806 09:24:59.245004 (+ 297us) server_negotiation.cc:299] Negotiation successful
0806 09:24:59.245255 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":360,"threads_started":1}
I20260806 09:24:59.246758 3558 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:24:59.345024 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 3368
I20260806 09:24:59.372181 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 2821
I20260806 09:24:59.373760 2904 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:24:59.506382 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 2821
I20260806 09:24:59.545024 16730 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20260806 09:24:59.891786 16730 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20260806 09:24:59.892031 16730 mini_ranger.cc:66] Stopping Ranger...
I20260806 09:25:00.249792 16730 mini_ranger.cc:68] Stopped Ranger
2026-08-06 09:25:00.250 UTC [1412] LOG: received smart shutdown request
2026-08-06 09:25:00.255 UTC [1412] LOG: background worker "logical replication launcher" (PID 1421) exited with exit code 1
2026-08-06 09:25:00.255 UTC [1416] LOG: shutting down
2026-08-06 09:25:00.462 UTC [1412] LOG: database system is shut down
2026-08-06T09:25:00Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (62855 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:25:00 dist-test-slave-h3ft krb5kdc[3669](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:25:00 dist-test-slave-h3ft krb5kdc[3669](info): set up 2 sockets
Aug 06 09:25:00 dist-test-slave-h3ft krb5kdc[3669](info): commencing operation
krb5kdc: starting...
W20260806 09:25:02.036455 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.387s user 0.002s sys 0.041s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:25:02 dist-test-slave-h3ft krb5kdc[3669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:25:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:25:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:25:02.422086 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41065
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:37193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:41065,127.16.86.189:43987,127.16.86.188:39189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:25:02.889159 3684 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:02.889573 3684 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:02.889755 3684 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:02.898334 3684 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:25:02.898505 3684 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:02.898589 3684 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:02.898689 3684 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:25:02.898767 3684 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:25:02.910907 3684 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:37193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:41065,127.16.86.189:43987,127.16.86.188:39189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:41065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:02.913061 3684 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:02.914882 3684 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:02.931813 3694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:02.932117 3693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:02.931800 3696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:02.932972 3684 server_base.cc:1034] running on GCE node
I20260806 09:25:02.934078 3684 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:02.936089 3684 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:02.937602 3684 hybrid_clock.cc:648] HybridClock initialized: now 1786008302937474 us; error 89 us; skew 500 ppm
I20260806 09:25:02.938455 3684 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:02 dist-test-slave-h3ft krb5kdc[3669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008302, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:02.947701 3684 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:25:02.949951 3684 webserver.cc:460] Webserver started at http://127.16.86.190:33537/ using document root <none> and password file <none>
I20260806 09:25:02.950862 3684 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:02.951073 3684 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:02.951532 3684 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:25:02.955380 3684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc"
format_stamp: "Formatted at 2026-08-06 09:25:02 on dist-test-slave-h3ft"
server_key: "fdb0cbe0196d7230d5f2901a4ce369c8"
server_key_iv: "e5be39b0f7de4c95544698ebc81263ed"
server_key_version: "encryptionkey@0"
I20260806 09:25:02.956453 3684 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc"
format_stamp: "Formatted at 2026-08-06 09:25:02 on dist-test-slave-h3ft"
server_key: "fdb0cbe0196d7230d5f2901a4ce369c8"
server_key_iv: "e5be39b0f7de4c95544698ebc81263ed"
server_key_version: "encryptionkey@0"
I20260806 09:25:02.964035 3684 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260806 09:25:02.969175 3704 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:02.971640 3684 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.003s
I20260806 09:25:02.972043 3684 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc"
format_stamp: "Formatted at 2026-08-06 09:25:02 on dist-test-slave-h3ft"
server_key: "fdb0cbe0196d7230d5f2901a4ce369c8"
server_key_iv: "e5be39b0f7de4c95544698ebc81263ed"
server_key_version: "encryptionkey@0"
I20260806 09:25:02.972334 3684 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:02.997421 3684 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:03.003082 3684 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:03.003463 3684 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:03.029181 3684 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:41065
I20260806 09:25:03.029299 3777 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:41065 every 8 connection(s)
I20260806 09:25:03.030965 3684 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:25:03.034505 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 3684
I20260806 09:25:03.034873 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:25:03.035945 16730 external_mini_cluster.cc:1428] Setting key d79ae1ca3347581affd8ba3066c943e2
I20260806 09:25:03.037494 3779 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:03.050930 3779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.16.86.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.189@KRBTEST.COM" created.
Aug 06 09:25:03 dist-test-slave-h3ft krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008302, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:03.088146 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.056444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41065 (local address 127.0.0.1:60428)
0806 09:25:03.057739 (+ 1295us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.057807 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:03.060423 (+ 2616us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:03.061551 (+ 1128us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:03.061559 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:03.062479 (+ 920us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:03.062916 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:03.062937 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.064816 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.064821 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:03.066249 (+ 1428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:03.066265 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.067460 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.067467 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:03.067657 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:03.068545 (+ 888us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:03.068582 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:03.075408 (+ 6826us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:03.081694 (+ 6286us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:03.081710 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:03.081724 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:03.082374 (+ 650us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:03.083207 (+ 833us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:03.083211 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:03.083214 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:03.083698 (+ 484us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:03.085286 (+ 1588us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:03.085295 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:03.085602 (+ 307us) client_negotiation.cc:769] Sending connection context
0806 09:25:03.086038 (+ 436us) client_negotiation.cc:240] Negotiation successful
0806 09:25:03.086624 (+ 586us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":877,"thread_start_us":553,"threads_started":1}
I20260806 09:25:03.088275 3785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.056445 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60428 (local address 127.16.86.190:41065)
0806 09:25:03.058879 (+ 2434us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:03.058889 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:03.058953 (+ 64us) server_negotiation.cc:407] Connection header received
0806 09:25:03.060939 (+ 1986us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:03.060947 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:03.061268 (+ 321us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:03.061543 (+ 275us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:03.063220 (+ 1677us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.064631 (+ 1411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.066426 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.067175 (+ 749us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.067426 (+ 251us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:03.075901 (+ 8475us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:03.075921 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:03.075954 (+ 33us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:03.076055 (+ 101us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:03.081389 (+ 5334us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:03.082616 (+ 1227us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:03.082631 (+ 15us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:03.082638 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:03.082980 (+ 342us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:03.083903 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:03.083912 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:03.083918 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:03.085046 (+ 1128us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:03.085337 (+ 291us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:03.086269 (+ 932us) server_negotiation.cc:299] Negotiation successful
0806 09:25:03.086940 (+ 671us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":164,"server-negotiator.queue_time_us":2147,"spinlock_wait_cycles":77568,"thread_start_us":1789,"threads_started":1}
I20260806 09:25:03.092149 3779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:25:03.093911 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.093305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47656)
0806 09:25:03.093563 (+ 258us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.093727 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
W20260806 09:25:03.094347 3707 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
I20260806 09:25:03.096789 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.096138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47660)
0806 09:25:03.096392 (+ 254us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.096552 (+ 160us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260806 09:25:03.097258 3779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:43987: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.189.keytab.
Entry for principal kudu/127.16.86.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.189.keytab.
I20260806 09:25:03.125005 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.189:43987
--webserver_interface=127.16.86.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.189.keytab
--principal=kudu/127.16.86.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:37193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:41065,127.16.86.189:43987,127.16.86.188:39189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:25:03.146816 3779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } attempt: 1
I20260806 09:25:03.148612 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.148019 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47664)
0806 09:25:03.148299 (+ 280us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.148439 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 09:25:03.150156 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.149670 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47680)
0806 09:25:03.149881 (+ 211us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.149963 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
W20260806 09:25:03.150476 3779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:43987: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
I20260806 09:25:03.218845 3779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } attempt: 2
I20260806 09:25:03.220687 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.220032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47692)
0806 09:25:03.220398 (+ 366us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.220530 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":144}
I20260806 09:25:03.221943 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.221531 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47694)
0806 09:25:03.221683 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.221781 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 09:25:03.222270 3779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:43987: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
I20260806 09:25:03.313625 3779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } attempt: 3
I20260806 09:25:03.315367 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.314688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47700)
0806 09:25:03.314976 (+ 288us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.315194 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":118}
I20260806 09:25:03.316609 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.316216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47704)
0806 09:25:03.316371 (+ 155us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.316473 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20260806 09:25:03.316890 3779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:43987: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
I20260806 09:25:03.460223 3779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } attempt: 4
I20260806 09:25:03.461696 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.461124 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47710)
0806 09:25:03.461448 (+ 324us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.461546 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":179}
I20260806 09:25:03.463258 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.462465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47724)
0806 09:25:03.462807 (+ 342us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.462975 (+ 168us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260806 09:25:03.463663 3779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:43987: Network error: Client connection negotiation failed: client connection to 127.16.86.189:43987: connect: Connection refused (error 111)
W20260806 09:25:03.585755 3790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:03.586323 3790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:03.586519 3790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:03.595849 3790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:25:03.596083 3790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:03.596213 3790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:03.596334 3790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:25:03.596438 3790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:25:03.608850 3790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:37193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:41065,127.16.86.189:43987,127.16.86.188:39189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.189.keytab
--principal=kudu/127.16.86.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.189:43987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.86.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:03.610916 3790 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:03.612789 3790 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:03.628167 3796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:03.628306 3799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:03.628237 3802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:03.634085 3790 server_base.cc:1034] running on GCE node
I20260806 09:25:03.635566 3790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:03.638243 3790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:03.639668 3790 hybrid_clock.cc:648] HybridClock initialized: now 1786008303639569 us; error 82 us; skew 500 ppm
I20260806 09:25:03.640280 3790 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:03 dist-test-slave-h3ft krb5kdc[3669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008303, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:03.645005 3790 init.cc:377] Logged in from keytab as kudu/127.16.86.189@KRBTEST.COM (short username kudu)
I20260806 09:25:03.648103 3790 webserver.cc:460] Webserver started at http://127.16.86.189:43039/ using document root <none> and password file <none>
I20260806 09:25:03.649281 3790 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:03.649492 3790 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:03.649909 3790 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:25:03.654230 3790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/data/instance:
uuid: "7e3a65c938564a7b920c28803232abed"
format_stamp: "Formatted at 2026-08-06 09:25:03 on dist-test-slave-h3ft"
server_key: "45ffabf470bd7c8789abe34b291af7d1"
server_key_iv: "0768d7240e2f43ba6fbaa1205a2dd832"
server_key_version: "encryptionkey@0"
I20260806 09:25:03.655573 3790 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/wal/instance:
uuid: "7e3a65c938564a7b920c28803232abed"
format_stamp: "Formatted at 2026-08-06 09:25:03 on dist-test-slave-h3ft"
server_key: "45ffabf470bd7c8789abe34b291af7d1"
server_key_iv: "0768d7240e2f43ba6fbaa1205a2dd832"
server_key_version: "encryptionkey@0"
I20260806 09:25:03.664551 3790 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260806 09:25:03.670377 3810 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:03.672876 3790 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260806 09:25:03.673202 3790 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/wal
uuid: "7e3a65c938564a7b920c28803232abed"
format_stamp: "Formatted at 2026-08-06 09:25:03 on dist-test-slave-h3ft"
server_key: "45ffabf470bd7c8789abe34b291af7d1"
server_key_iv: "0768d7240e2f43ba6fbaa1205a2dd832"
server_key_version: "encryptionkey@0"
I20260806 09:25:03.673520 3790 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:03.702787 3790 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:03.709748 3790 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:03.710193 3790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:03.738111 3790 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.189:43987
I20260806 09:25:03.738186 3882 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.189:43987 every 8 connection(s)
I20260806 09:25:03.740442 3790 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
I20260806 09:25:03.746008 3886 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:03.747834 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 3790
I20260806 09:25:03.748095 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-1/wal/instance
I20260806 09:25:03.748785 16730 external_mini_cluster.cc:1428] Setting key 6fd581de5a9756ada381c9610330ddfb
I20260806 09:25:03.758554 3886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:25:03.763029 3779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } attempt: 5
Aug 06 09:25:03 dist-test-slave-h3ft krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008302, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for kudu/127.16.86.189@KRBTEST.COM
Aug 06 09:25:03 dist-test-slave-h3ft krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008303, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.189@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.86.188@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.188@KRBTEST.COM" created.
I20260806 09:25:03.794852 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.763786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47738)
0806 09:25:03.764145 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.764172 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:03.764346 (+ 174us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:03.770985 (+ 6639us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:03.770994 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:03.771861 (+ 867us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:03.772344 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:03.772365 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.776105 (+ 3740us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.776114 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:03.777646 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:03.777666 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.779045 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.779052 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:03.779334 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:03.780045 (+ 711us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:03.780076 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:03.783671 (+ 3595us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:03.789766 (+ 6095us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:03.789774 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:03.789778 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:03.790624 (+ 846us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:03.791839 (+ 1215us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:03.791847 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:03.791852 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:03.792036 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:03.793476 (+ 1440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:03.793483 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:03.793668 (+ 185us) client_negotiation.cc:769] Sending connection context
0806 09:25:03.793888 (+ 220us) client_negotiation.cc:240] Negotiation successful
0806 09:25:03.794327 (+ 439us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":205}
I20260806 09:25:03.796295 3890 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.762931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60430 (local address 127.16.86.190:41065)
0806 09:25:03.766812 (+ 3881us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:03.766822 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:03.768487 (+ 1665us) server_negotiation.cc:407] Connection header received
0806 09:25:03.770530 (+ 2043us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:03.770537 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:03.770654 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:03.770795 (+ 141us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:03.773201 (+ 2406us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.774075 (+ 874us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.776446 (+ 2371us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.777457 (+ 1011us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.777642 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:03.785634 (+ 7992us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:03.785647 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:03.785653 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:03.785735 (+ 82us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:03.789987 (+ 4252us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:03.791550 (+ 1563us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:03.791558 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:03.791562 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:03.791797 (+ 235us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:03.792610 (+ 813us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:03.792615 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:03.792617 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:03.793045 (+ 428us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:03.793221 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:03.794069 (+ 848us) server_negotiation.cc:299] Negotiation successful
0806 09:25:03.795802 (+ 1733us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1400,"server-negotiator.queue_time_us":3674,"thread_start_us":3478,"threads_started":1}
I20260806 09:25:03.796794 3894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.765277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47738 (local address 127.16.86.189:43987)
0806 09:25:03.766525 (+ 1248us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:03.766543 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:03.766607 (+ 64us) server_negotiation.cc:407] Connection header received
0806 09:25:03.770063 (+ 3456us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:03.770149 (+ 86us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:03.770604 (+ 455us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:03.772815 (+ 2211us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:03.773285 (+ 470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.775826 (+ 2541us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.777859 (+ 2033us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.778837 (+ 978us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.778986 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:03.784056 (+ 5070us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:03.784070 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:03.784092 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:03.784147 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:03.789310 (+ 5163us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:03.791169 (+ 1859us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:03.791187 (+ 18us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:03.791199 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:03.791587 (+ 388us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:03.792180 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:03.792189 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:03.792196 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:03.793298 (+ 1102us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:03.793556 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:03.794455 (+ 899us) server_negotiation.cc:299] Negotiation successful
0806 09:25:03.794852 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":85,"server-negotiator.queue_time_us":882,"thread_start_us":586,"threads_started":1}
I20260806 09:25:03.796942 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.763970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41065 (local address 127.0.0.1:60430)
0806 09:25:03.767959 (+ 3989us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.768042 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:03.770090 (+ 2048us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:03.770906 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:03.770920 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:03.771860 (+ 940us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:03.772942 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:03.772968 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.774279 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.774284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:03.776224 (+ 1940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:03.776245 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.777782 (+ 1537us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.777790 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:03.778166 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:03.779726 (+ 1560us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:03.779755 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:03.784880 (+ 5125us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:03.790223 (+ 5343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:03.790249 (+ 26us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:03.790267 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:03.791318 (+ 1051us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:03.792233 (+ 915us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:03.792238 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:03.792242 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:03.792446 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:03.793231 (+ 785us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:03.793245 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:03.793536 (+ 291us) client_negotiation.cc:769] Sending connection context
0806 09:25:03.793901 (+ 365us) client_negotiation.cc:240] Negotiation successful
0806 09:25:03.794697 (+ 796us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3569,"mutex_wait_us":4,"thread_start_us":3055,"threads_started":1}
I20260806 09:25:03.799440 3886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:25:03.800863 3779 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:25:03.802326 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.801683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47012)
0806 09:25:03.801936 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.802074 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 09:25:03.803663 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.803260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47014)
0806 09:25:03.803427 (+ 167us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.803515 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20260806 09:25:03.804101 3779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:39189: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Aug 06 09:25:03 dist-test-slave-h3ft krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008303, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.189@KRBTEST.COM for kudu/127.16.86.189@KRBTEST.COM
I20260806 09:25:03.818156 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.800669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47748)
0806 09:25:03.801285 (+ 616us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.801308 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:03.801465 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:03.802006 (+ 541us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:03.802011 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:03.802625 (+ 614us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:03.802880 (+ 255us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:03.802893 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.804460 (+ 1567us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.804465 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:03.805857 (+ 1392us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:03.805879 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.806739 (+ 860us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.806749 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:03.807175 (+ 426us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:03.807984 (+ 809us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:03.808014 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:03.811521 (+ 3507us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:03.815508 (+ 3987us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:03.815514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:03.815518 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:03.815950 (+ 432us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:03.816343 (+ 393us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:03.816349 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:03.816353 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:03.816500 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:03.817071 (+ 571us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:03.817075 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:03.817226 (+ 151us) client_negotiation.cc:769] Sending connection context
0806 09:25:03.817372 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 09:25:03.817705 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"mutex_wait_us":28}
I20260806 09:25:03.818164 3894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.801019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47748 (local address 127.16.86.189:43987)
0806 09:25:03.801280 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:03.801287 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:03.801365 (+ 78us) server_negotiation.cc:407] Connection header received
0806 09:25:03.801618 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:03.801630 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:03.801692 (+ 62us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:03.801811 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:03.803113 (+ 1302us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.804290 (+ 1177us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.806074 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:03.806543 (+ 469us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:03.806734 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:03.811807 (+ 5073us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:03.811813 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:03.811816 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:03.811849 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:03.815337 (+ 3488us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:03.816084 (+ 747us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:03.816090 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:03.816093 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:03.816230 (+ 137us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:03.816618 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:03.816628 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:03.816630 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:03.816951 (+ 321us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:03.817119 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:03.817464 (+ 345us) server_negotiation.cc:299] Negotiation successful
0806 09:25:03.817860 (+ 396us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":172,"server-negotiator.queue_time_us":119}
I20260806 09:25:03.819468 3886 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:25:03.821364 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.820474 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47030)
0806 09:25:03.820766 (+ 292us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.821059 (+ 293us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20260806 09:25:03.821868 3811 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
I20260806 09:25:03.824164 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.823480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47044)
0806 09:25:03.823764 (+ 284us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.823934 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260806 09:25:03.824486 3886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:39189: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.188.keytab.
Entry for principal kudu/127.16.86.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.188.keytab.
I20260806 09:25:03.840396 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.188:39189
--webserver_interface=127.16.86.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.188.keytab
--principal=kudu/127.16.86.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:37193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:41065,127.16.86.189:43987,127.16.86.188:39189
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:25:03.855459 3779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } attempt: 1
I20260806 09:25:03.857558 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.856579 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47058)
0806 09:25:03.856952 (+ 373us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.857207 (+ 255us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":125}
I20260806 09:25:03.859350 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.858715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47068)
0806 09:25:03.858952 (+ 237us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.859132 (+ 180us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260806 09:25:03.859724 3779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:39189: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
I20260806 09:25:03.874002 3886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } attempt: 1
I20260806 09:25:03.876019 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.875186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47076)
0806 09:25:03.875573 (+ 387us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.875747 (+ 174us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
I20260806 09:25:03.877413 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.876909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47086)
0806 09:25:03.877096 (+ 187us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.877186 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 09:25:03.877719 3886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:39189: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
I20260806 09:25:03.928076 3779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } attempt: 2
I20260806 09:25:03.929986 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.929165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47090)
0806 09:25:03.929526 (+ 361us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.929704 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":121}
I20260806 09:25:03.931922 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.931346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47098)
0806 09:25:03.931576 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.931708 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
W20260806 09:25:03.932287 3779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:39189: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
I20260806 09:25:03.945974 3886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } attempt: 2
I20260806 09:25:03.947654 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.947045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47114)
0806 09:25:03.947314 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.947445 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 09:25:03.948832 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:03.948413 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47120)
0806 09:25:03.948569 (+ 156us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:03.948724 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20260806 09:25:03.949054 3886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:39189: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
I20260806 09:25:04.038596 3779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } attempt: 3
I20260806 09:25:04.040277 3886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } attempt: 3
I20260806 09:25:04.040562 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.039665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47122)
0806 09:25:04.040057 (+ 392us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.040238 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":124}
I20260806 09:25:04.041749 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.040982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47136)
0806 09:25:04.041328 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.041490 (+ 162us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":146}
I20260806 09:25:04.042043 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.041477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47142)
0806 09:25:04.041733 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.041846 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":103}
W20260806 09:25:04.042402 3779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:39189: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
I20260806 09:25:04.043397 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.042727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47148)
0806 09:25:04.043161 (+ 434us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.043255 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":213}
W20260806 09:25:04.043720 3886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:39189: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
I20260806 09:25:04.186959 3886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } attempt: 4
I20260806 09:25:04.188275 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.187764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47150)
0806 09:25:04.187962 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.188073 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20260806 09:25:04.193724 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.193101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47160)
0806 09:25:04.193406 (+ 305us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.193533 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
W20260806 09:25:04.194303 3886 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:39189: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
I20260806 09:25:04.219672 3779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } attempt: 4
I20260806 09:25:04.221051 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.220504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47166)
0806 09:25:04.220744 (+ 240us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.220856 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 09:25:04.222563 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.222060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47172)
0806 09:25:04.222315 (+ 255us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.222421 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
W20260806 09:25:04.222903 3779 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:39189: Network error: Client connection negotiation failed: client connection to 127.16.86.188:39189: connect: Connection refused (error 111)
W20260806 09:25:04.301302 3898 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:04.301746 3898 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:04.301857 3898 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:04.310127 3898 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:25:04.310314 3898 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:04.310369 3898 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:04.310427 3898 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:25:04.310482 3898 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:25:04.322312 3898 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:37193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:41065,127.16.86.189:43987,127.16.86.188:39189
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.188.keytab
--principal=kudu/127.16.86.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.188:39189
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.86.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:04.324234 3898 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:04.326123 3898 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:04.342736 3905 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:04.342952 3908 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:04.342968 3906 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:04.343617 3898 server_base.cc:1034] running on GCE node
I20260806 09:25:04.344890 3898 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:04.346698 3898 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:04.347937 3898 hybrid_clock.cc:648] HybridClock initialized: now 1786008304347829 us; error 114 us; skew 500 ppm
I20260806 09:25:04.348413 3898 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:04 dist-test-slave-h3ft krb5kdc[3669](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008304, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:04.352134 3898 init.cc:377] Logged in from keytab as kudu/127.16.86.188@KRBTEST.COM (short username kudu)
I20260806 09:25:04.354058 3898 webserver.cc:460] Webserver started at http://127.16.86.188:41627/ using document root <none> and password file <none>
I20260806 09:25:04.354830 3898 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:04.354943 3898 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:04.355316 3898 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:25:04.358632 3898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/data/instance:
uuid: "562d191510144e24b03ff5f8168bede1"
format_stamp: "Formatted at 2026-08-06 09:25:04 on dist-test-slave-h3ft"
server_key: "fee3ca7b86cb6a615951846f710d092c"
server_key_iv: "764a5a5a23ef1a5478a76eef4c0ce71f"
server_key_version: "encryptionkey@0"
I20260806 09:25:04.359681 3898 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/wal/instance:
uuid: "562d191510144e24b03ff5f8168bede1"
format_stamp: "Formatted at 2026-08-06 09:25:04 on dist-test-slave-h3ft"
server_key: "fee3ca7b86cb6a615951846f710d092c"
server_key_iv: "764a5a5a23ef1a5478a76eef4c0ce71f"
server_key_version: "encryptionkey@0"
I20260806 09:25:04.366153 3898 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:25:04.371343 3918 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:04.373531 3898 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 09:25:04.373835 3898 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/wal
uuid: "562d191510144e24b03ff5f8168bede1"
format_stamp: "Formatted at 2026-08-06 09:25:04 on dist-test-slave-h3ft"
server_key: "fee3ca7b86cb6a615951846f710d092c"
server_key_iv: "764a5a5a23ef1a5478a76eef4c0ce71f"
server_key_version: "encryptionkey@0"
I20260806 09:25:04.374095 3898 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:04.400794 3898 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:04.405978 3898 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:04.406344 3898 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:04.431835 3898 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.188:39189
I20260806 09:25:04.431998 3993 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.188:39189 every 8 connection(s)
I20260806 09:25:04.433569 3898 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
I20260806 09:25:04.439214 3994 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:04.442798 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 3898
I20260806 09:25:04.443121 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACert.1786008061536593-16730-0/minicluster-data/master-2/wal/instance
I20260806 09:25:04.443809 16730 external_mini_cluster.cc:1428] Setting key d4c9e051ace1404b737bae455b272306
I20260806 09:25:04.450891 3994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:25:04 dist-test-slave-h3ft krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.188@KRBTEST.COM
Aug 06 09:25:04 dist-test-slave-h3ft krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008304, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.188@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:04.475783 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.454576 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60444 (local address 127.16.86.190:41065)
0806 09:25:04.456325 (+ 1749us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:04.456331 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:04.456355 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:25:04.456431 (+ 76us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:04.456438 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:04.456534 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:04.456639 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:04.459092 (+ 2453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.460207 (+ 1115us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.462679 (+ 2472us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.463184 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.463295 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.468362 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:04.468371 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:04.468377 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:04.468449 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:04.471931 (+ 3482us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.473095 (+ 1164us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.473101 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.473104 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.473291 (+ 187us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.473827 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.473831 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.473833 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.474155 (+ 322us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:04.474324 (+ 169us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:04.475106 (+ 782us) server_negotiation.cc:299] Negotiation successful
0806 09:25:04.475347 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1566,"thread_start_us":1431,"threads_started":1}
I20260806 09:25:04.477483 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.446799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47176 (local address 127.16.86.188:39189)
0806 09:25:04.448186 (+ 1387us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:04.448205 (+ 19us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:04.448270 (+ 65us) server_negotiation.cc:407] Connection header received
0806 09:25:04.450420 (+ 2150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:04.450451 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:04.451178 (+ 727us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:04.451667 (+ 489us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:04.454176 (+ 2509us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.456339 (+ 2163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.459243 (+ 2904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.460565 (+ 1322us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.460750 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.467372 (+ 6622us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:04.467388 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:04.467414 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:04.467479 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:04.471329 (+ 3850us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.472433 (+ 1104us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.472472 (+ 39us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.472475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.472716 (+ 241us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.473250 (+ 534us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.473253 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.473255 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.474170 (+ 915us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:04.474556 (+ 386us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:04.474924 (+ 368us) server_negotiation.cc:299] Negotiation successful
0806 09:25:04.475519 (+ 595us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":1,"server-negotiator.queue_time_us":1019,"thread_start_us":556,"threads_started":1}
I20260806 09:25:04.477484 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.454477 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:41065 (local address 127.0.0.1:60444)
0806 09:25:04.455496 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.455561 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:04.455763 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:04.456874 (+ 1111us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:04.456885 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:04.458188 (+ 1303us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:04.458725 (+ 537us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:04.458744 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.460465 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.460470 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:04.462422 (+ 1952us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:04.462471 (+ 49us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.463507 (+ 1036us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.463517 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:04.463756 (+ 239us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.464621 (+ 865us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:04.464663 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:04.467962 (+ 3299us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:04.472146 (+ 4184us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:04.472159 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:04.472173 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:04.472903 (+ 730us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:04.473432 (+ 529us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:04.473437 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:04.473440 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:04.473624 (+ 184us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:04.474312 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:04.474321 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:04.474569 (+ 248us) client_negotiation.cc:769] Sending connection context
0806 09:25:04.474927 (+ 358us) client_negotiation.cc:240] Negotiation successful
0806 09:25:04.475656 (+ 729us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":672,"mutex_wait_us":190,"thread_start_us":395,"threads_started":1}
I20260806 09:25:04.480356 3994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:25:04 dist-test-slave-h3ft krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008304, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.188@KRBTEST.COM for kudu/127.16.86.189@KRBTEST.COM
Aug 06 09:25:04 dist-test-slave-h3ft krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:04.493603 3886 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } attempt: 5
I20260806 09:25:04.499563 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.481508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:43987 (local address 127.0.0.1:47760)
0806 09:25:04.481787 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.481818 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:04.482002 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:04.482721 (+ 719us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:04.482731 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:04.483349 (+ 618us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:04.483649 (+ 300us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:04.483664 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.484871 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.484875 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:04.486168 (+ 1293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:04.486186 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.487060 (+ 874us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.487068 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:04.487340 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.488096 (+ 756us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:04.488115 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:04.491122 (+ 3007us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:04.496134 (+ 5012us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:04.496141 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:04.496146 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:04.496789 (+ 643us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:04.497316 (+ 527us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:04.497321 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:04.497326 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:04.497530 (+ 204us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:04.498246 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:04.498252 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:04.498499 (+ 247us) client_negotiation.cc:769] Sending connection context
0806 09:25:04.498695 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 09:25:04.499149 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":91}
I20260806 09:25:04.500124 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.481402 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47760 (local address 127.16.86.189:43987)
0806 09:25:04.482290 (+ 888us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:04.482297 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:04.482322 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:25:04.482396 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:04.482402 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:04.482511 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:04.482657 (+ 146us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:04.483868 (+ 1211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.484689 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.486356 (+ 1667us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.486812 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.486917 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.491415 (+ 4498us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:04.491425 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:04.491431 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:04.491488 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:04.495965 (+ 4477us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.496926 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.496934 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.496940 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.497159 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.497651 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.497657 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.497660 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.498069 (+ 409us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:04.498251 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:04.498855 (+ 604us) server_negotiation.cc:299] Negotiation successful
0806 09:25:04.499236 (+ 381us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":727,"thread_start_us":524,"threads_started":1}
I20260806 09:25:04.500262 3779 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } attempt: 5
I20260806 09:25:04.501750 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.482521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60448 (local address 127.16.86.190:41065)
0806 09:25:04.482706 (+ 185us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:04.482719 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:04.482739 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:25:04.482804 (+ 65us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:04.482812 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:04.482895 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:04.482979 (+ 84us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:04.484710 (+ 1731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.485455 (+ 745us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.487275 (+ 1820us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.487743 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.487925 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.493636 (+ 5711us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:04.493643 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:04.493648 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:04.493700 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:04.497507 (+ 3807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.498484 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.498489 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.498493 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.498660 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.499158 (+ 498us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.499163 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.499166 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.499547 (+ 381us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:04.499876 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:04.501102 (+ 1226us) server_negotiation.cc:299] Negotiation successful
0806 09:25:04.501383 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:25:04.501740 3994 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:25:04 dist-test-slave-h3ft krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008303, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.189@KRBTEST.COM for kudu/127.16.86.188@KRBTEST.COM
Aug 06 09:25:04 dist-test-slave-h3ft krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008302, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for kudu/127.16.86.188@KRBTEST.COM
Aug 06 09:25:04 dist-test-slave-h3ft krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008302, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.189@KRBTEST.COM
I20260806 09:25:04.518783 3892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.494487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47180)
0806 09:25:04.494717 (+ 230us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.494745 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:04.494884 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:04.495292 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:04.495297 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:04.496015 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:04.496296 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:04.496312 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.497954 (+ 1642us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.497960 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:04.499330 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:04.499345 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.500748 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.500754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:04.500965 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.501788 (+ 823us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:04.501813 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:04.506466 (+ 4653us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:04.514923 (+ 8457us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:04.514933 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:04.514941 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:04.515643 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:04.516266 (+ 623us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:04.516272 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:04.516275 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:04.516477 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:04.517344 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:04.517350 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:04.517579 (+ 229us) client_negotiation.cc:769] Sending connection context
0806 09:25:04.517741 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 09:25:04.518163 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":75}
I20260806 09:25:04.520321 3998 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.494625 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47180 (local address 127.16.86.188:39189)
0806 09:25:04.494874 (+ 249us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:04.494881 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:04.494916 (+ 35us) server_negotiation.cc:407] Connection header received
0806 09:25:04.495060 (+ 144us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:04.495070 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:04.495140 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:04.495256 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:04.496511 (+ 1255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.497774 (+ 1263us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.499553 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.500361 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.500522 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.506757 (+ 6235us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:04.506764 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:04.506768 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:04.506824 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:04.512602 (+ 5778us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.515877 (+ 3275us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.515882 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.515886 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.516102 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.516666 (+ 564us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.516673 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.516679 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.517187 (+ 508us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:04.519346 (+ 2159us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:04.519519 (+ 173us) server_negotiation.cc:299] Negotiation successful
0806 09:25:04.519871 (+ 352us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":105}
Aug 06 09:25:04 dist-test-slave-h3ft krb5kdc[3669](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008304, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.188@KRBTEST.COM for kudu/127.16.86.188@KRBTEST.COM
I20260806 09:25:04.527837 3784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.500904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47194)
0806 09:25:04.501302 (+ 398us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.501324 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:04.501423 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:04.504622 (+ 3199us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:04.504628 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:04.505146 (+ 518us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:04.505411 (+ 265us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:04.505427 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.509460 (+ 4033us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.509469 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:04.511085 (+ 1616us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:04.511104 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.512257 (+ 1153us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.512263 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:04.512453 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.513486 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:04.513514 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:04.516525 (+ 3011us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:04.524296 (+ 7771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:04.524303 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:04.524307 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:04.525009 (+ 702us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:04.525613 (+ 604us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:04.525618 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:04.525622 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:04.525813 (+ 191us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:04.526631 (+ 818us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:04.526637 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:04.526847 (+ 210us) client_negotiation.cc:769] Sending connection context
0806 09:25:04.527052 (+ 205us) client_negotiation.cc:240] Negotiation successful
0806 09:25:04.527412 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":247}
I20260806 09:25:04.529155 3886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed: Bootstrap starting.
I20260806 09:25:04.529695 4005 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.501765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47194 (local address 127.16.86.188:39189)
0806 09:25:04.503043 (+ 1278us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:04.503050 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:04.503074 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:25:04.503140 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:04.503144 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:04.504429 (+ 1285us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:04.507148 (+ 2719us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:04.507494 (+ 346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.509273 (+ 1779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.511307 (+ 2034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.512063 (+ 756us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.512842 (+ 779us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.516865 (+ 4023us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:04.516871 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:04.516876 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:04.516950 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:04.524063 (+ 7113us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.525178 (+ 1115us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.525183 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.525187 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.525462 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.525966 (+ 504us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.525971 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.525976 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.526401 (+ 425us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:04.526551 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:04.528792 (+ 2241us) server_negotiation.cc:299] Negotiation successful
0806 09:25:04.529090 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1079,"spinlock_wait_cycles":3456,"thread_start_us":838,"threads_started":1}
I20260806 09:25:04.530556 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.504026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47776 (local address 127.16.86.189:43987)
0806 09:25:04.504360 (+ 334us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:04.504368 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:04.504387 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:25:04.504444 (+ 57us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:04.504449 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:04.504554 (+ 105us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:04.504644 (+ 90us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:04.508090 (+ 3446us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.509193 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.511308 (+ 2115us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.512016 (+ 708us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.512842 (+ 826us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.519645 (+ 6803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:04.519654 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:04.519659 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:04.519715 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:04.525384 (+ 5669us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.526554 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.526562 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.526566 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.526745 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.527363 (+ 618us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.527368 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.527372 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.527877 (+ 505us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:04.529601 (+ 1724us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:04.529816 (+ 215us) server_negotiation.cc:299] Negotiation successful
0806 09:25:04.530115 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":193}
I20260806 09:25:04.534847 3886 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:04.536386 3886 log.cc:824] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:04.536794 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:25:04.537254 3779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc: Bootstrap starting.
I20260806 09:25:04.540294 3886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed: No bootstrap required, opened a new log
I20260806 09:25:04.542315 3779 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:04.542830 3999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.503043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:39189 (local address 127.0.0.1:47204)
0806 09:25:04.503389 (+ 346us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:04.503407 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:04.505763 (+ 2356us) spinlock_profiling.cc:243] Waited 1.31 ms on lock 0x612000023328. stack: 00007f932589666e 00007f9325896430 00007f932491d5ed 00000000008ea064 00000000008ea02d 00000000008e99fa 00007f932588c0fd 00007f932588bd1e 00007f93387993cd 00007f933879936c 00007f9338765b3b 00007f932c29b536 00007f932c299d9e 00007f932c321dca 00007f932c31ff68 00007f932c35382f
0806 09:25:04.505813 (+ 50us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:04.514898 (+ 9085us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:04.514903 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:04.515555 (+ 652us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:04.515867 (+ 312us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:04.515881 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.517858 (+ 1977us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.517863 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:04.519674 (+ 1811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:04.519692 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.521200 (+ 1508us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.521206 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:04.521453 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.522147 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:04.522193 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:04.526180 (+ 3987us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:04.539193 (+ 13013us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:04.539206 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:04.539212 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:04.539998 (+ 786us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:04.540556 (+ 558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:04.540560 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:04.540563 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:04.540730 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:04.541582 (+ 852us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:04.541587 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:04.541748 (+ 161us) client_negotiation.cc:769] Sending connection context
0806 09:25:04.541918 (+ 170us) client_negotiation.cc:240] Negotiation successful
0806 09:25:04.542343 (+ 425us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":248,"spinlock_wait_cycles":3004288}
I20260806 09:25:04.544230 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.503416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47204 (local address 127.16.86.188:39189)
0806 09:25:04.513608 (+ 10192us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:04.513945 (+ 337us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:04.513982 (+ 37us) server_negotiation.cc:407] Connection header received
0806 09:25:04.514048 (+ 66us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:04.514055 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:04.514270 (+ 215us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:04.514452 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:04.516132 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.517638 (+ 1506us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.519887 (+ 2249us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.520733 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.520918 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.526461 (+ 5543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:04.526473 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:04.526479 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:04.526526 (+ 47us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:04.531315 (+ 4789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.540205 (+ 8890us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.540213 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.540216 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.540415 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.540857 (+ 442us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.540862 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.540865 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.541344 (+ 479us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:04.541525 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:04.543483 (+ 1958us) server_negotiation.cc:299] Negotiation successful
0806 09:25:04.543762 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10004,"thread_start_us":9801,"threads_started":1}
I20260806 09:25:04.544430 3779 log.cc:824] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:04.551542 3994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1: Bootstrap starting.
I20260806 09:25:04.554194 3886 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } }
I20260806 09:25:04.555037 3886 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:04.555244 3886 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7e3a65c938564a7b920c28803232abed, State: Initialized, Role: FOLLOWER
I20260806 09:25:04.555966 3779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc: No bootstrap required, opened a new log
I20260806 09:25:04.556447 3886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } }
I20260806 09:25:04.557168 3994 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:04.559242 3994 log.cc:824] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:04.571270 3994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1: No bootstrap required, opened a new log
I20260806 09:25:04.574164 3779 raft_consensus.cc:348] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } }
I20260806 09:25:04.574708 3779 raft_consensus.cc:374] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:04.574874 3779 raft_consensus.cc:729] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fa7362e6cc1c4c3f98c5f1e9495d27bc, State: Initialized, Role: FOLLOWER
I20260806 09:25:04.575851 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } } }
I20260806 09:25:04.576485 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:25:04.577617 3886 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:25:04.577685 3779 consensus_queue.cc:260] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } }
I20260806 09:25:04.585346 4022 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } } }
I20260806 09:25:04.585912 4022 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:25:04.588351 3779 sys_catalog.cc:564] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:25:04.588727 3994 raft_consensus.cc:348] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } }
I20260806 09:25:04.589578 3994 raft_consensus.cc:374] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:04.589808 3994 raft_consensus.cc:729] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 562d191510144e24b03ff5f8168bede1, State: Initialized, Role: FOLLOWER
I20260806 09:25:04.598618 3994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } }
I20260806 09:25:04.601696 4022 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:25:04.602051 4022 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } }
I20260806 09:25:04.603994 4022 leader_election.cc:288] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7e3a65c938564a7b920c28803232abed (127.16.86.189:43987), 562d191510144e24b03ff5f8168bede1 (127.16.86.188:39189)
I20260806 09:25:04.606187 3849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3a65c938564a7b920c28803232abed" is_pre_election: true
I20260806 09:25:04.606928 3849 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fa7362e6cc1c4c3f98c5f1e9495d27bc in term 0.
I20260806 09:25:04.607910 3707 leader_election.cc:302] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e3a65c938564a7b920c28803232abed, fa7362e6cc1c4c3f98c5f1e9495d27bc; no voters:
I20260806 09:25:04.608515 4022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:25:04.608683 4022 raft_consensus.cc:482] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:25:04.608783 4022 raft_consensus.cc:3037] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:04.614109 4022 raft_consensus.cc:504] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } }
I20260806 09:25:04.616390 3849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3a65c938564a7b920c28803232abed"
I20260806 09:25:04.616676 3849 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:04.620850 3849 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7362e6cc1c4c3f98c5f1e9495d27bc in term 1.
I20260806 09:25:04.621675 3707 leader_election.cc:302] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7e3a65c938564a7b920c28803232abed, fa7362e6cc1c4c3f98c5f1e9495d27bc; no voters:
I20260806 09:25:04.628799 4022 leader_election.cc:288] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [CANDIDATE]: Term 1 election: Requested vote from peers 7e3a65c938564a7b920c28803232abed (127.16.86.189:43987), 562d191510144e24b03ff5f8168bede1 (127.16.86.188:39189)
I20260806 09:25:04.630573 4022 raft_consensus.cc:2781] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:25:04.631765 4022 raft_consensus.cc:686] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [term 1 LEADER]: Becoming Leader. State: Replica: fa7362e6cc1c4c3f98c5f1e9495d27bc, State: Running, Role: LEADER
I20260806 09:25:04.633024 4031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } } }
I20260806 09:25:04.636549 4031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:25:04.637063 4031 raft_consensus.cc:482] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:25:04.637566 4031 raft_consensus.cc:504] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } }
I20260806 09:25:04.638619 4001 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.575083 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60452 (local address 127.16.86.190:41065)
0806 09:25:04.575317 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:04.575324 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:04.596065 (+ 20741us) server_negotiation.cc:407] Connection header received
0806 09:25:04.596806 (+ 741us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:04.596816 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:04.596939 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:04.597037 (+ 98us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:04.605380 (+ 8343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.606514 (+ 1134us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.611198 (+ 4684us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.611978 (+ 780us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.612100 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.622422 (+ 10322us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:04.622434 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:04.622443 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:04.622502 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:04.627302 (+ 4800us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.628416 (+ 1114us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.628421 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.628425 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.628628 (+ 203us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.634842 (+ 6214us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.634852 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.634857 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.635401 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:04.635733 (+ 332us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:04.637806 (+ 2073us) server_negotiation.cc:299] Negotiation successful
0806 09:25:04.638144 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":91}
I20260806 09:25:04.636521 4022 consensus_queue.cc:237] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } }
I20260806 09:25:04.643977 4002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.582084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47790 (local address 127.16.86.189:43987)
0806 09:25:04.582277 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:04.582284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:04.601095 (+ 18811us) server_negotiation.cc:407] Connection header received
0806 09:25:04.602096 (+ 1001us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:04.602101 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:04.602210 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:04.602298 (+ 88us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:04.608481 (+ 6183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.609643 (+ 1162us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.617876 (+ 8233us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.618603 (+ 727us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.618713 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.628952 (+ 10239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:04.628960 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:04.628966 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:04.629026 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:04.633540 (+ 4514us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.636927 (+ 3387us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.636934 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.636937 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.637138 (+ 201us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.638930 (+ 1792us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.638938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.638943 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.639548 (+ 605us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:04.639897 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:04.643245 (+ 3348us) server_negotiation.cc:299] Negotiation successful
0806 09:25:04.643568 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20260806 09:25:04.645289 3743 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "562d191510144e24b03ff5f8168bede1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" is_pre_election: true
I20260806 09:25:04.650655 3994 sys_catalog.cc:564] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:25:04.650771 3959 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562d191510144e24b03ff5f8168bede1"
I20260806 09:25:04.651588 3959 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:04.656867 3959 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fa7362e6cc1c4c3f98c5f1e9495d27bc in term 1.
I20260806 09:25:04.658427 3960 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "562d191510144e24b03ff5f8168bede1" is_pre_election: true
I20260806 09:25:04.659075 3960 raft_consensus.cc:2362] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate fa7362e6cc1c4c3f98c5f1e9495d27bc in term 1. Re-sending same reply.
W20260806 09:25:04.659569 3925 tablet.cc:2369] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:25:04.663555 4031 leader_election.cc:288] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fa7362e6cc1c4c3f98c5f1e9495d27bc (127.16.86.190:41065), 7e3a65c938564a7b920c28803232abed (127.16.86.189:43987)
I20260806 09:25:04.665089 3849 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "562d191510144e24b03ff5f8168bede1" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7e3a65c938564a7b920c28803232abed" is_pre_election: true
I20260806 09:25:04.665287 4033 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa7362e6cc1c4c3f98c5f1e9495d27bc. Latest consensus state: current_term: 1 leader_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } } }
I20260806 09:25:04.665776 4033 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:04.665781 3849 raft_consensus.cc:2379] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 562d191510144e24b03ff5f8168bede1 in current term 1: Already voted for candidate fa7362e6cc1c4c3f98c5f1e9495d27bc in this term.
I20260806 09:25:04.667311 3921 leader_election.cc:302] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 562d191510144e24b03ff5f8168bede1; no voters: 7e3a65c938564a7b920c28803232abed, fa7362e6cc1c4c3f98c5f1e9495d27bc
I20260806 09:25:04.667979 4031 raft_consensus.cc:2726] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 09:25:04.674467 4007 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:04.591618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47214 (local address 127.16.86.188:39189)
0806 09:25:04.591911 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:04.591922 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:04.601695 (+ 9773us) server_negotiation.cc:407] Connection header received
0806 09:25:04.601970 (+ 275us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:04.601981 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:04.602153 (+ 172us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:04.602317 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:04.607537 (+ 5220us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.609097 (+ 1560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.615352 (+ 6255us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:04.616431 (+ 1079us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:04.616597 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:04.651760 (+ 35163us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:04.651774 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:04.651783 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:04.651904 (+ 121us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:04.658842 (+ 6938us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.670336 (+ 11494us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.670347 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.670354 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.670671 (+ 317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:04.671568 (+ 897us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:04.671576 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:04.671582 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:04.672248 (+ 666us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:04.672686 (+ 438us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:04.673667 (+ 981us) server_negotiation.cc:299] Negotiation successful
0806 09:25:04.674072 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 09:25:04.679467 4040 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:25:04.686959 4060 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:25:04.693686 4040 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:25:04.690631 3790 master_runner.cc:186] Error getting master registration for 127.16.86.190:41065: OK, instance_id { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" instance_seqno: 1786008303022044 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:25:04.702647 3790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 09:25:04.714226 3820 tablet.cc:2369] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:25:04.729888 3959 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [term 1 FOLLOWER]: Refusing update from remote peer fa7362e6cc1c4c3f98c5f1e9495d27bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:25:04.708869 3898 master_runner.cc:186] Error getting master registration for 127.16.86.190:41065: OK, instance_id { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" instance_seqno: 1786008303022044 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:25:04.732718 3898 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 09:25:04.733928 3849 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [term 1 FOLLOWER]: Refusing update from remote peer fa7362e6cc1c4c3f98c5f1e9495d27bc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260806 09:25:04.736179 4075 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260806 09:25:04.736054 4033 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 09:25:04.736351 4075 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:25:04.744129 4033 consensus_queue.cc:1035] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 09:25:04.758672 3710 tablet.cc:2369] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:25:04.767868 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa7362e6cc1c4c3f98c5f1e9495d27bc. Latest consensus state: current_term: 1 leader_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } } }
I20260806 09:25:04.768249 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:25:04.772950 4079 mvcc.cc:205] Tried to move back new op lower bound from 7315490016158011392 to 7315490015815045120. Current Snapshot: MvccSnapshot[applied={T|T < 7315490016158011392}]
I20260806 09:25:04.775539 4022 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } } }
I20260806 09:25:04.776084 4022 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:04.778517 4040 catalog_manager.cc:1347] Generated new cluster ID: 4852aad661ea45929ae35f4a420f59a3
I20260806 09:25:04.778692 4040 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:25:04.783221 4087 mvcc.cc:205] Tried to move back new op lower bound from 7315490016158011392 to 7315490015815045120. Current Snapshot: MvccSnapshot[applied={T|T < 7315490016158011392}]
I20260806 09:25:04.776926 4031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader fa7362e6cc1c4c3f98c5f1e9495d27bc. Latest consensus state: current_term: 1 leader_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } } }
I20260806 09:25:04.792217 4031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:25:04.801769 4022 sys_catalog.cc:455] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } } }
I20260806 09:25:04.802233 4022 sys_catalog.cc:458] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:04.803262 4014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } } }
I20260806 09:25:04.803517 4014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:25:04.809089 4031 sys_catalog.cc:455] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fa7362e6cc1c4c3f98c5f1e9495d27bc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 41065 } } peers { permanent_uuid: "7e3a65c938564a7b920c28803232abed" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 43987 } } peers { permanent_uuid: "562d191510144e24b03ff5f8168bede1" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 39189 } } }
I20260806 09:25:04.809402 4031 sys_catalog.cc:458] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:25:04.820279 4040 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:25:04.824287 4040 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:25:04.843231 4040 catalog_manager.cc:5868] T 00000000000000000000000000000000 P fa7362e6cc1c4c3f98c5f1e9495d27bc: Generated new TSK 0
I20260806 09:25:04.844558 4040 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:25:05.693825 4060 catalog_manager.cc:1259] Loaded cluster ID: 4852aad661ea45929ae35f4a420f59a3
I20260806 09:25:05.694244 4060 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed: loading cluster ID for follower catalog manager: success
I20260806 09:25:05.698177 4060 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed: acquiring CA information for follower catalog manager: success
I20260806 09:25:05.701792 4060 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 7e3a65c938564a7b920c28803232abed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:25:05.739585 4075 catalog_manager.cc:1259] Loaded cluster ID: 4852aad661ea45929ae35f4a420f59a3
I20260806 09:25:05.739777 4075 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1: loading cluster ID for follower catalog manager: success
I20260806 09:25:05.743199 4075 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1: acquiring CA information for follower catalog manager: success
I20260806 09:25:05.745882 4075 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 562d191510144e24b03ff5f8168bede1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:25:05.956223 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 3684
I20260806 09:25:05.957924 3772 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:06.093405 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 3684
W20260806 09:25:06.124925 3816 connection.cc:535] server connection from 127.0.0.1:47738 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 09:25:06.125686 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 3790
I20260806 09:25:06.127348 3877 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:06.280371 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 3790
I20260806 09:25:06.316854 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 3898
I20260806 09:25:06.318352 3987 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:06.447175 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 3898
2026-08-06T09:25:06Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5859 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:25:06 dist-test-slave-h3ft krb5kdc[4107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:25:06 dist-test-slave-h3ft krb5kdc[4107](info): set up 2 sockets
Aug 06 09:25:06 dist-test-slave-h3ft krb5kdc[4107](info): commencing operation
krb5kdc: starting...
W20260806 09:25:07.918079 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.409s user 0.000s sys 0.049s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:25:08 dist-test-slave-h3ft krb5kdc[4107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:25:08Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:25:08Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:25:08.312539 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:34523
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44007
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:34523
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:25:08.784296 4128 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:08.784795 4128 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:08.784996 4128 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:08.794129 4128 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:25:08.794348 4128 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:08.794484 4128 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:08.794637 4128 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:25:08.794735 4128 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:25:08.809397 4128 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44007
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:34523
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:34523
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:08.811890 4128 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:08.814059 4128 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:08.830521 4142 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:08.830521 4135 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:08.831166 4139 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:08.832084 4128 server_base.cc:1034] running on GCE node
I20260806 09:25:08.833159 4128 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:08.834910 4128 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:08.836323 4128 hybrid_clock.cc:648] HybridClock initialized: now 1786008308836254 us; error 64 us; skew 500 ppm
I20260806 09:25:08.836930 4128 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:08 dist-test-slave-h3ft krb5kdc[4107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008308, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:08.847075 4128 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:25:08.849998 4128 webserver.cc:460] Webserver started at http://127.16.86.190:33591/ using document root <none> and password file <none>
I20260806 09:25:08.851410 4128 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:08.851621 4128 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:08.852010 4128 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:25:08.855772 4128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "8b8c53103a634fb5aa5a7e5e1ce1fd76"
format_stamp: "Formatted at 2026-08-06 09:25:08 on dist-test-slave-h3ft"
server_key: "e062eefd7134b3b18d62ab28fd165779"
server_key_iv: "5ac6eb1563c14a20b3263b21cce67ef3"
server_key_version: "encryptionkey@0"
I20260806 09:25:08.856664 4128 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "8b8c53103a634fb5aa5a7e5e1ce1fd76"
format_stamp: "Formatted at 2026-08-06 09:25:08 on dist-test-slave-h3ft"
server_key: "e062eefd7134b3b18d62ab28fd165779"
server_key_iv: "5ac6eb1563c14a20b3263b21cce67ef3"
server_key_version: "encryptionkey@0"
I20260806 09:25:08.864456 4128 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.004s sys 0.003s
I20260806 09:25:08.870115 4150 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:08.873612 4128 fs_manager.cc:725] Time spent opening block manager: real 0.006s user 0.001s sys 0.001s
I20260806 09:25:08.873884 4128 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "8b8c53103a634fb5aa5a7e5e1ce1fd76"
format_stamp: "Formatted at 2026-08-06 09:25:08 on dist-test-slave-h3ft"
server_key: "e062eefd7134b3b18d62ab28fd165779"
server_key_iv: "5ac6eb1563c14a20b3263b21cce67ef3"
server_key_version: "encryptionkey@0"
I20260806 09:25:08.874194 4128 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:08.894622 4128 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:08.900681 4128 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:08.901057 4128 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:08.929401 4128 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:34523
I20260806 09:25:08.929611 4221 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:34523 every 8 connection(s)
I20260806 09:25:08.931116 4128 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:25:08.934399 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 4128
I20260806 09:25:08.934777 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:25:08.935845 16730 external_mini_cluster.cc:1428] Setting key ca48c4d75b1e999ba7488102d73c7d53
I20260806 09:25:08.937405 4222 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:25:08 dist-test-slave-h3ft krb5kdc[4107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008308, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:08.960693 4222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76: Bootstrap starting.
I20260806 09:25:08.966320 4222 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:08.968292 4222 log.cc:824] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:08.973937 4222 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76: No bootstrap required, opened a new log
I20260806 09:25:08.975617 4227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:08.938823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34776 (local address 127.16.86.190:34523)
0806 09:25:08.940978 (+ 2155us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:08.940994 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:08.941047 (+ 53us) server_negotiation.cc:407] Connection header received
0806 09:25:08.943377 (+ 2330us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:08.943405 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:08.943922 (+ 517us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:08.944503 (+ 581us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:08.947718 (+ 3215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:08.949969 (+ 2251us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:08.952961 (+ 2992us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:08.953975 (+ 1014us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:08.954234 (+ 259us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:08.961803 (+ 7569us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:08.961816 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:08.961840 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:08.961915 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:08.967561 (+ 5646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:08.969572 (+ 2011us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:08.969584 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:08.969595 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:08.969924 (+ 329us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:08.970623 (+ 699us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:08.970633 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:08.970640 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:08.971819 (+ 1179us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:08.972278 (+ 459us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:08.972923 (+ 645us) server_negotiation.cc:299] Negotiation successful
0806 09:25:08.973631 (+ 708us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1715,"thread_start_us":1030,"threads_started":1}
I20260806 09:25:08.981312 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:25:08.985610 4222 raft_consensus.cc:348] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8c53103a634fb5aa5a7e5e1ce1fd76" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34523 } }
I20260806 09:25:08.986145 4222 raft_consensus.cc:374] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:08.986308 4222 raft_consensus.cc:729] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8b8c53103a634fb5aa5a7e5e1ce1fd76, State: Initialized, Role: FOLLOWER
I20260806 09:25:08.987201 4222 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8c53103a634fb5aa5a7e5e1ce1fd76" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34523 } }
I20260806 09:25:08.987526 4222 raft_consensus.cc:388] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:25:08.987713 4222 raft_consensus.cc:482] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:25:08.987943 4222 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:08.990904 4222 raft_consensus.cc:504] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8c53103a634fb5aa5a7e5e1ce1fd76" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34523 } }
I20260806 09:25:08.991768 4222 leader_election.cc:302] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8b8c53103a634fb5aa5a7e5e1ce1fd76; no voters:
I20260806 09:25:08.992651 4222 leader_election.cc:288] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:25:08.992914 4229 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:25:08.994032 4229 raft_consensus.cc:686] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [term 1 LEADER]: Becoming Leader. State: Replica: 8b8c53103a634fb5aa5a7e5e1ce1fd76, State: Running, Role: LEADER
I20260806 09:25:08.994702 4229 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8c53103a634fb5aa5a7e5e1ce1fd76" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34523 } }
I20260806 09:25:08.995677 4222 sys_catalog.cc:564] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:25:08.998703 4234 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8b8c53103a634fb5aa5a7e5e1ce1fd76. Latest consensus state: current_term: 1 leader_uuid: "8b8c53103a634fb5aa5a7e5e1ce1fd76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8c53103a634fb5aa5a7e5e1ce1fd76" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34523 } } }
I20260806 09:25:08.998608 4230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8b8c53103a634fb5aa5a7e5e1ce1fd76" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8b8c53103a634fb5aa5a7e5e1ce1fd76" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34523 } } }
I20260806 09:25:08.999321 4234 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:08.999469 4230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76 [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:09.006775 4239 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:25:09.012315 4239 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:25:09.026063 4239 catalog_manager.cc:1347] Generated new cluster ID: 680cbf5b2cc2440198f840f02920d435
I20260806 09:25:09.026319 4239 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:25:09.038165 4239 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:25:09.040607 4239 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:25:09.056595 4239 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 8b8c53103a634fb5aa5a7e5e1ce1fd76: Generated new TSK 0
I20260806 09:25:09.057739 4239 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:25:09.098709 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 4128
I20260806 09:25:09.100737 4216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:09.214847 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 4128
2026-08-06T09:25:09Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2760 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:25:09 dist-test-slave-h3ft krb5kdc[4262](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:25:09 dist-test-slave-h3ft krb5kdc[4262](info): set up 2 sockets
Aug 06 09:25:09 dist-test-slave-h3ft krb5kdc[4262](info): commencing operation
krb5kdc: starting...
W20260806 09:25:10.642521 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.372s user 0.001s sys 0.056s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:25:10 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:25:10Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:25:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:25:11.021318 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:33769
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:33769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:25:11.445658 4278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:11.446499 4278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:11.446753 4278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:11.456370 4278 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:25:11.456579 4278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:11.456732 4278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:11.456875 4278 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:25:11.456962 4278 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:25:11.469676 4278 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:33769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:33769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:11.471704 4278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:11.473651 4278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:11.490180 4284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:11.490392 4285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:11.490408 4288 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:11.491755 4278 server_base.cc:1034] running on GCE node
I20260806 09:25:11.492998 4278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:11.495159 4278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:11.496490 4278 hybrid_clock.cc:648] HybridClock initialized: now 1786008311496430 us; error 48 us; skew 500 ppm
I20260806 09:25:11.497011 4278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:11 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008311, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:11.501384 4278 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:25:11.503614 4278 webserver.cc:460] Webserver started at http://127.16.86.190:33311/ using document root <none> and password file <none>
I20260806 09:25:11.504554 4278 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:11.504742 4278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:11.505125 4278 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:25:11.509035 4278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "00198bb056544817a977c392a45892ab"
format_stamp: "Formatted at 2026-08-06 09:25:11 on dist-test-slave-h3ft"
server_key: "9438fd7507bb5ab45c90bd984cf9a708"
server_key_iv: "843373a7cb1769d9935403164bbb9b67"
server_key_version: "encryptionkey@0"
I20260806 09:25:11.510022 4278 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "00198bb056544817a977c392a45892ab"
format_stamp: "Formatted at 2026-08-06 09:25:11 on dist-test-slave-h3ft"
server_key: "9438fd7507bb5ab45c90bd984cf9a708"
server_key_iv: "843373a7cb1769d9935403164bbb9b67"
server_key_version: "encryptionkey@0"
I20260806 09:25:11.516755 4278 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260806 09:25:11.521354 4300 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:11.523579 4278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:25:11.523871 4278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "00198bb056544817a977c392a45892ab"
format_stamp: "Formatted at 2026-08-06 09:25:11 on dist-test-slave-h3ft"
server_key: "9438fd7507bb5ab45c90bd984cf9a708"
server_key_iv: "843373a7cb1769d9935403164bbb9b67"
server_key_version: "encryptionkey@0"
I20260806 09:25:11.524151 4278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:11.545553 4278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:11.551108 4278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:11.551479 4278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:11.574874 4278 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:33769
I20260806 09:25:11.574930 4369 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:33769 every 8 connection(s)
I20260806 09:25:11.576763 4278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:25:11.580734 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 4278
I20260806 09:25:11.581017 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:25:11.582088 16730 external_mini_cluster.cc:1428] Setting key be12d75f2d91709e76ba97b266d38d22
I20260806 09:25:11.582056 4370 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:11.596756 4370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Bootstrap starting.
Aug 06 09:25:11 dist-test-slave-h3ft krb5kdc[4262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:11.600222 4370 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:11.601275 4370 log.cc:824] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:11.605988 4370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: No bootstrap required, opened a new log
I20260806 09:25:11.612448 4374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:11.584897 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48404 (local address 127.16.86.190:33769)
0806 09:25:11.586313 (+ 1416us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:11.586335 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:11.586398 (+ 63us) server_negotiation.cc:407] Connection header received
0806 09:25:11.588503 (+ 2105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:11.588535 (+ 32us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:11.589112 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:11.589651 (+ 539us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:11.591677 (+ 2026us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:11.593238 (+ 1561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:11.594827 (+ 1589us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:11.595733 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:11.595968 (+ 235us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:11.601599 (+ 5631us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:11.601611 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:11.601633 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:11.601685 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:11.606331 (+ 4646us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:11.607416 (+ 1085us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:11.607426 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:11.607437 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:11.607720 (+ 283us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:11.608142 (+ 422us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:11.608147 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:11.608150 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:11.609102 (+ 952us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:11.609501 (+ 399us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:11.609960 (+ 459us) server_negotiation.cc:299] Negotiation successful
0806 09:25:11.610602 (+ 642us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1091,"thread_start_us":672,"threads_started":1}
I20260806 09:25:11.617714 4370 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } }
I20260806 09:25:11.618237 4370 raft_consensus.cc:374] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:11.618377 4370 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00198bb056544817a977c392a45892ab, State: Initialized, Role: FOLLOWER
I20260806 09:25:11.619199 4370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } }
I20260806 09:25:11.619451 4370 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:25:11.619561 4370 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:25:11.619717 4370 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:11.622243 4370 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } }
I20260806 09:25:11.622933 4370 leader_election.cc:302] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00198bb056544817a977c392a45892ab; no voters:
I20260806 09:25:11.623793 4370 leader_election.cc:288] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:25:11.624018 4377 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:25:11.626365 4377 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 1 LEADER]: Becoming Leader. State: Replica: 00198bb056544817a977c392a45892ab, State: Running, Role: LEADER
I20260806 09:25:11.626457 4370 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:25:11.627175 4377 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } }
I20260806 09:25:11.631129 4380 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "00198bb056544817a977c392a45892ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } } }
I20260806 09:25:11.631639 4380 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:11.631338 4382 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00198bb056544817a977c392a45892ab. Latest consensus state: current_term: 1 leader_uuid: "00198bb056544817a977c392a45892ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } } }
I20260806 09:25:11.631888 4382 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:11.635142 4391 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:25:11.641606 4391 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:25:11.654464 4391 catalog_manager.cc:1347] Generated new cluster ID: e083fe5589f54e90b261b02584efec5f
I20260806 09:25:11.654632 4391 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:25:11.667706 4391 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:25:11.669497 4391 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:25:11.679446 4391 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Generated new TSK 0
I20260806 09:25:11.680395 4391 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:25:11.775758 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:33769
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:25:12.215255 4406 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:12.215672 4406 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:12.215790 4406 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:12.223985 4406 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:12.224251 4406 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:12.224470 4406 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:25:12.237145 4406 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:33769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:12.239099 4406 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:12.241030 4406 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:12.259195 4411 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:12.259198 4416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:12.259258 4414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:12.260500 4406 server_base.cc:1034] running on GCE node
I20260806 09:25:12.261490 4406 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:12.262876 4406 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:12.264192 4406 hybrid_clock.cc:648] HybridClock initialized: now 1786008312264117 us; error 65 us; skew 500 ppm
I20260806 09:25:12.264690 4406 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:12 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008312, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:12.269152 4406 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:25:12.271498 4406 webserver.cc:460] Webserver started at http://127.16.86.129:34609/ using document root <none> and password file <none>
I20260806 09:25:12.272488 4406 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:12.272686 4406 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:12.273104 4406 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:25:12.277243 4406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "26e3a0be0611487c8dc56ee0e0863f6d"
format_stamp: "Formatted at 2026-08-06 09:25:12 on dist-test-slave-h3ft"
server_key: "85b70fa9fb01dd7214f5065609bdcc67"
server_key_iv: "c887b9883a683eb02be0d45df5c2d44a"
server_key_version: "encryptionkey@0"
I20260806 09:25:12.278386 4406 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "26e3a0be0611487c8dc56ee0e0863f6d"
format_stamp: "Formatted at 2026-08-06 09:25:12 on dist-test-slave-h3ft"
server_key: "85b70fa9fb01dd7214f5065609bdcc67"
server_key_iv: "c887b9883a683eb02be0d45df5c2d44a"
server_key_version: "encryptionkey@0"
I20260806 09:25:12.286098 4406 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260806 09:25:12.291515 4426 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:12.293607 4406 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:25:12.293865 4406 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "26e3a0be0611487c8dc56ee0e0863f6d"
format_stamp: "Formatted at 2026-08-06 09:25:12 on dist-test-slave-h3ft"
server_key: "85b70fa9fb01dd7214f5065609bdcc67"
server_key_iv: "c887b9883a683eb02be0d45df5c2d44a"
server_key_version: "encryptionkey@0"
I20260806 09:25:12.294143 4406 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:12.318648 4406 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:12.324131 4406 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:12.324558 4406 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:12.325776 4406 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:25:12.328202 4406 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:25:12.328328 4406 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:12.328441 4406 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:25:12.328526 4406 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:12.371076 4406 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:39089
I20260806 09:25:12.371273 4580 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:39089 every 8 connection(s)
I20260806 09:25:12.372759 4406 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:25:12.380362 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 4406
I20260806 09:25:12.380673 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:25:12.381469 16730 external_mini_cluster.cc:1428] Setting key af9d2583d12bf7583edf2c7c2397e64d
Aug 06 09:25:12 dist-test-slave-h3ft krb5kdc[4262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008312, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:12.400408 4583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:12.376661 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:55227 (local address 127.16.86.190:33769)
0806 09:25:12.377264 (+ 603us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:12.377270 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:12.378696 (+ 1426us) server_negotiation.cc:407] Connection header received
0806 09:25:12.381154 (+ 2458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:12.381160 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:12.381291 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:12.381433 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:12.383475 (+ 2042us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:12.384413 (+ 938us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:12.387143 (+ 2730us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:12.387621 (+ 478us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:12.387742 (+ 121us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:12.393262 (+ 5520us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:12.393268 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:12.393271 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:12.393319 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:12.396376 (+ 3057us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:12.397379 (+ 1003us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:12.397384 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:12.397387 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:12.397525 (+ 138us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:12.398062 (+ 537us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:12.398067 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:12.398070 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:12.398395 (+ 325us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:12.398571 (+ 176us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:12.399629 (+ 1058us) server_negotiation.cc:299] Negotiation successful
0806 09:25:12.399935 (+ 306us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":439,"thread_start_us":285,"threads_started":1}
I20260806 09:25:12.401657 4584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:12.377401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:55227)
0806 09:25:12.378419 (+ 1018us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:12.378484 (+ 65us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:12.380832 (+ 2348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:12.381712 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:12.381722 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:12.382354 (+ 632us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:12.383246 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:12.383283 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:12.384603 (+ 1320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:12.384610 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:12.386899 (+ 2289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:12.386920 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:12.387909 (+ 989us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:12.387918 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:12.388293 (+ 375us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:12.389426 (+ 1133us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:12.389464 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:12.392901 (+ 3437us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:12.396583 (+ 3682us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:12.396597 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:12.396628 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:12.397210 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:12.397650 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:12.397656 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:12.397660 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:12.397924 (+ 264us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:12.398537 (+ 613us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:12.398547 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:12.399141 (+ 594us) client_negotiation.cc:769] Sending connection context
0806 09:25:12.399475 (+ 334us) client_negotiation.cc:240] Negotiation successful
0806 09:25:12.400182 (+ 707us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":693,"thread_start_us":380,"threads_started":1}
I20260806 09:25:12.403760 4581 heartbeater.cc:347] Connected to a master server at 127.16.86.190:33769
I20260806 09:25:12.404175 4581 heartbeater.cc:464] Registering TS with master...
I20260806 09:25:12.405119 4581 heartbeater.cc:511] Master 127.16.86.190:33769 requested a full tablet report, sending...
I20260806 09:25:12.408159 4326 ts_manager.cc:194] Registered new tserver with Master: 26e3a0be0611487c8dc56ee0e0863f6d (127.16.86.129:39089)
I20260806 09:25:12.410633 4326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:55227
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:25:12.459483 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:33769
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:25:12.840111 4590 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:12.840430 4590 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:12.840531 4590 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:12.852197 4590 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:12.852382 4590 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:12.852624 4590 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:25:12.863595 4590 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:33769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:12.864966 4590 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:12.866320 4590 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:12.881858 4598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:12.881924 4599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:12.882154 4601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:12.883414 4590 server_base.cc:1034] running on GCE node
I20260806 09:25:12.884397 4590 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:12.885860 4590 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:12.887182 4590 hybrid_clock.cc:648] HybridClock initialized: now 1786008312887106 us; error 67 us; skew 500 ppm
I20260806 09:25:12.887616 4590 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:12 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008312, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:12.891772 4590 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:25:12.893965 4590 webserver.cc:460] Webserver started at http://127.16.86.130:44299/ using document root <none> and password file <none>
I20260806 09:25:12.894906 4590 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:12.895131 4590 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:12.895556 4590 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:25:12.899782 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "42e16385638a4d5b9655f89d5877a4c7"
format_stamp: "Formatted at 2026-08-06 09:25:12 on dist-test-slave-h3ft"
server_key: "cf283098b82044fbb6a46ce61fa8ecca"
server_key_iv: "684ef4a1e5f7e50fdcd91b22f0e49dc6"
server_key_version: "encryptionkey@0"
I20260806 09:25:12.900913 4590 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "42e16385638a4d5b9655f89d5877a4c7"
format_stamp: "Formatted at 2026-08-06 09:25:12 on dist-test-slave-h3ft"
server_key: "cf283098b82044fbb6a46ce61fa8ecca"
server_key_iv: "684ef4a1e5f7e50fdcd91b22f0e49dc6"
server_key_version: "encryptionkey@0"
I20260806 09:25:12.908450 4590 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260806 09:25:12.913455 4614 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:12.915474 4590 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:25:12.915711 4590 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "42e16385638a4d5b9655f89d5877a4c7"
format_stamp: "Formatted at 2026-08-06 09:25:12 on dist-test-slave-h3ft"
server_key: "cf283098b82044fbb6a46ce61fa8ecca"
server_key_iv: "684ef4a1e5f7e50fdcd91b22f0e49dc6"
server_key_version: "encryptionkey@0"
I20260806 09:25:12.915982 4590 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:12.941437 4590 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:12.947403 4590 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:12.947790 4590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:12.949119 4590 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:25:12.950881 4590 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:25:12.951005 4590 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:12.951200 4590 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:25:12.951256 4590 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:12.993065 4590 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:34417
I20260806 09:25:12.993183 4758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:34417 every 8 connection(s)
I20260806 09:25:12.994760 4590 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:25:12.998397 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 4590
I20260806 09:25:12.998626 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:25:12.999467 16730 external_mini_cluster.cc:1428] Setting key e5021ab2920a6ed19c8e46cc3582c6e0
Aug 06 09:25:13 dist-test-slave-h3ft krb5kdc[4262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008312, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:13.022601 4763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:12.997931 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:44689 (local address 127.16.86.190:33769)
0806 09:25:12.998645 (+ 714us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:12.998654 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:12.999916 (+ 1262us) server_negotiation.cc:407] Connection header received
0806 09:25:13.002330 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:13.002334 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:13.002415 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:13.002535 (+ 120us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:13.005322 (+ 2787us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.006185 (+ 863us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.009047 (+ 2862us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.009535 (+ 488us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.009696 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:13.015392 (+ 5696us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:13.015398 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:13.015402 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:13.015444 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:13.018662 (+ 3218us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:13.019679 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:13.019684 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:13.019686 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:13.019827 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:13.020459 (+ 632us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:13.020463 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:13.020465 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:13.020791 (+ 326us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:13.020979 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:13.021862 (+ 883us) server_negotiation.cc:299] Negotiation successful
0806 09:25:13.022134 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":524,"thread_start_us":322,"threads_started":1}
I20260806 09:25:13.023947 4764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:12.998419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:44689)
0806 09:25:12.999654 (+ 1235us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:12.999731 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:13.001871 (+ 2140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:13.002923 (+ 1052us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:13.002940 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:13.003959 (+ 1019us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:13.005060 (+ 1101us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:13.005095 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.006386 (+ 1291us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.006394 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:13.008837 (+ 2443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:13.008856 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.009862 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.009873 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:13.010269 (+ 396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:13.011356 (+ 1087us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:13.011395 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:13.014981 (+ 3586us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:13.018853 (+ 3872us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:13.018866 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:13.018888 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:13.019523 (+ 635us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:13.019985 (+ 462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:13.019991 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:13.019995 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:13.020294 (+ 299us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:13.020948 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:13.020957 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:13.021450 (+ 493us) client_negotiation.cc:769] Sending connection context
0806 09:25:13.021754 (+ 304us) client_negotiation.cc:240] Negotiation successful
0806 09:25:13.022407 (+ 653us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":807,"thread_start_us":485,"threads_started":1}
I20260806 09:25:13.025838 4759 heartbeater.cc:347] Connected to a master server at 127.16.86.190:33769
I20260806 09:25:13.026238 4759 heartbeater.cc:464] Registering TS with master...
I20260806 09:25:13.027309 4759 heartbeater.cc:511] Master 127.16.86.190:33769 requested a full tablet report, sending...
I20260806 09:25:13.029469 4326 ts_manager.cc:194] Registered new tserver with Master: 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417)
I20260806 09:25:13.030953 4326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:44689
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:25:13.089030 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:33769
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:25:13.415205 4581 heartbeater.cc:503] Master 127.16.86.190:33769 was elected leader, sending a full tablet report...
W20260806 09:25:13.518916 4768 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:13.519354 4768 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:13.519471 4768 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:13.528424 4768 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:13.528584 4768 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:13.528717 4768 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:25:13.540587 4768 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:33769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:13.542351 4768 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:13.544112 4768 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:13.561537 4776 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:13.561376 4779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:13.561535 4781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:13.562870 4768 server_base.cc:1034] running on GCE node
I20260806 09:25:13.563704 4768 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:13.564915 4768 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:13.566172 4768 hybrid_clock.cc:648] HybridClock initialized: now 1786008313566112 us; error 52 us; skew 500 ppm
I20260806 09:25:13.566578 4768 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:13 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008313, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:13.570652 4768 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:25:13.572734 4768 webserver.cc:460] Webserver started at http://127.16.86.131:40153/ using document root <none> and password file <none>
I20260806 09:25:13.573585 4768 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:13.573748 4768 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:13.574105 4768 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:25:13.577869 4768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "633faae4693e45bcbe46babba1396b2a"
format_stamp: "Formatted at 2026-08-06 09:25:13 on dist-test-slave-h3ft"
server_key: "a140f6feb89dfebe83412ba1df2547c3"
server_key_iv: "eca9f49c0ad008dfe4c2ddbaadf25095"
server_key_version: "encryptionkey@0"
I20260806 09:25:13.578933 4768 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "633faae4693e45bcbe46babba1396b2a"
format_stamp: "Formatted at 2026-08-06 09:25:13 on dist-test-slave-h3ft"
server_key: "a140f6feb89dfebe83412ba1df2547c3"
server_key_iv: "eca9f49c0ad008dfe4c2ddbaadf25095"
server_key_version: "encryptionkey@0"
I20260806 09:25:13.585384 4768 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260806 09:25:13.590095 4792 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:13.591931 4768 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:25:13.592154 4768 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "633faae4693e45bcbe46babba1396b2a"
format_stamp: "Formatted at 2026-08-06 09:25:13 on dist-test-slave-h3ft"
server_key: "a140f6feb89dfebe83412ba1df2547c3"
server_key_iv: "eca9f49c0ad008dfe4c2ddbaadf25095"
server_key_version: "encryptionkey@0"
I20260806 09:25:13.592401 4768 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:13.613893 4768 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:13.618676 4768 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:13.619045 4768 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:13.620371 4768 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:25:13.622066 4768 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:25:13.622154 4768 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:13.622303 4768 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:25:13.622361 4768 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:13.664656 4768 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:38173
I20260806 09:25:13.664695 4930 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:38173 every 8 connection(s)
I20260806 09:25:13.666106 4768 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:25:13.671742 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 4768
I20260806 09:25:13.671957 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:25:13.672488 16730 external_mini_cluster.cc:1428] Setting key 8b6adcd492b7d494a96b018bf50f6de9
Aug 06 09:25:13 dist-test-slave-h3ft krb5kdc[4262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008313, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:13.693779 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:13.669734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:50589 (local address 127.16.86.190:33769)
0806 09:25:13.670426 (+ 692us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:13.670434 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:13.671992 (+ 1558us) server_negotiation.cc:407] Connection header received
0806 09:25:13.674973 (+ 2981us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:13.674978 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:13.675122 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:13.675272 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:13.677919 (+ 2647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.678755 (+ 836us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.680694 (+ 1939us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.681099 (+ 405us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.681208 (+ 109us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:13.685427 (+ 4219us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:13.685432 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:13.685436 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:13.685475 (+ 39us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:13.689155 (+ 3680us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:13.690600 (+ 1445us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:13.690607 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:13.690612 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:13.690865 (+ 253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:13.691625 (+ 760us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:13.691629 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:13.691631 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:13.692011 (+ 380us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:13.692178 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:13.693140 (+ 962us) server_negotiation.cc:299] Negotiation successful
0806 09:25:13.693395 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":514,"thread_start_us":327,"threads_started":1}
I20260806 09:25:13.694674 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:13.670459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:50589)
0806 09:25:13.671605 (+ 1146us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:13.671667 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:13.674563 (+ 2896us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:13.675677 (+ 1114us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:13.675694 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:13.676714 (+ 1020us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:13.677699 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:13.677723 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.678988 (+ 1265us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.678992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:13.680511 (+ 1519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:13.680527 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.681340 (+ 813us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.681345 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:13.681590 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:13.682362 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:13.682393 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:13.685137 (+ 2744us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:13.689365 (+ 4228us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:13.689378 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:13.689405 (+ 27us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:13.690366 (+ 961us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:13.691120 (+ 754us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:13.691126 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:13.691129 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:13.691447 (+ 318us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:13.692168 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:13.692177 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:13.692620 (+ 443us) client_negotiation.cc:769] Sending connection context
0806 09:25:13.692899 (+ 279us) client_negotiation.cc:240] Negotiation successful
0806 09:25:13.693404 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":769,"thread_start_us":439,"threads_started":1}
I20260806 09:25:13.696781 4931 heartbeater.cc:347] Connected to a master server at 127.16.86.190:33769
I20260806 09:25:13.697227 4931 heartbeater.cc:464] Registering TS with master...
I20260806 09:25:13.698305 4931 heartbeater.cc:511] Master 127.16.86.190:33769 requested a full tablet report, sending...
I20260806 09:25:13.700528 4326 ts_manager.cc:194] Registered new tserver with Master: 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173)
I20260806 09:25:13.701721 4326 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:50589
I20260806 09:25:13.703794 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:25:13.725236 4937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:13.709492 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48410 (local address 127.16.86.190:33769)
0806 09:25:13.709725 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:13.709730 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:13.710349 (+ 619us) server_negotiation.cc:407] Connection header received
0806 09:25:13.710516 (+ 167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:13.710520 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:13.710599 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:13.710723 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:13.712492 (+ 1769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.713317 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.715193 (+ 1876us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.715605 (+ 412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.715725 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:13.718927 (+ 3202us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:13.718932 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:13.718936 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:13.718972 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:13.721969 (+ 2997us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:13.722942 (+ 973us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:13.722945 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:13.722948 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:13.723216 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:13.723671 (+ 455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:13.723674 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:13.723677 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:13.724024 (+ 347us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:13.724260 (+ 236us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:13.724739 (+ 479us) server_negotiation.cc:299] Negotiation successful
0806 09:25:13.724951 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:25:13.739374 4326 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48410:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:25:13.746965 4326 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:25:13.777825 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:13.765102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49432 (local address 127.16.86.129:39089)
0806 09:25:13.767999 (+ 2897us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:13.768014 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:13.768058 (+ 44us) server_negotiation.cc:407] Connection header received
0806 09:25:13.768220 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:13.768233 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:13.768569 (+ 336us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:13.768893 (+ 324us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:13.770236 (+ 1343us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.771764 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.774586 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.776425 (+ 1839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.776602 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:13.776919 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:13.777221 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 09:25:13.777429 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2676,"thread_start_us":2485,"threads_started":1}
I20260806 09:25:13.777858 4956 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:13.764165 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39089 (local address 127.0.0.1:49432)
0806 09:25:13.765464 (+ 1299us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:13.765526 (+ 62us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:13.765746 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:13.769263 (+ 3517us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:13.769278 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:13.769319 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:13.769944 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:13.769979 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.771916 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.771922 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:13.774283 (+ 2361us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:13.774308 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.776644 (+ 2336us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.776650 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:13.776853 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:13.776932 (+ 79us) client_negotiation.cc:769] Sending connection context
0806 09:25:13.777128 (+ 196us) client_negotiation.cc:240] Negotiation successful
0806 09:25:13.777388 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":985,"thread_start_us":1882,"threads_started":1}
I20260806 09:25:13.778658 4952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:13.764349 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:34417 (local address 127.0.0.1:42758)
0806 09:25:13.766429 (+ 2080us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:13.766457 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:13.766659 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:13.770279 (+ 3620us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:13.770296 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:13.770331 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:13.770798 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:13.770812 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.773187 (+ 2375us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.773194 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:13.775734 (+ 2540us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:13.775752 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.777635 (+ 1883us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.777643 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:13.777853 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:13.777896 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 09:25:13.777969 (+ 73us) client_negotiation.cc:240] Negotiation successful
0806 09:25:13.778127 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1899,"mutex_wait_us":239,"thread_start_us":652,"threads_started":1}
I20260806 09:25:13.779403 4957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:13.764635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:38173 (local address 127.0.0.1:34890)
0806 09:25:13.769226 (+ 4591us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:13.769250 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:13.769401 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:13.770345 (+ 944us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:13.770352 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:13.770396 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:13.770985 (+ 589us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:13.771029 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.773189 (+ 2160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.773194 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:13.776220 (+ 3026us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:13.776236 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.778453 (+ 2217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.778461 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:13.778684 (+ 223us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:13.778725 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:25:13.778815 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 09:25:13.778962 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4427,"mutex_wait_us":80,"thread_start_us":4252,"threads_started":1}
I20260806 09:25:13.780218 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:13.765740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34890 (local address 127.16.86.131:38173)
0806 09:25:13.768513 (+ 2773us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:13.768522 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:13.769497 (+ 975us) server_negotiation.cc:407] Connection header received
0806 09:25:13.769613 (+ 116us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:13.769620 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:13.769842 (+ 222us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:13.770052 (+ 210us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:13.771227 (+ 1175us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.773011 (+ 1784us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.776416 (+ 3405us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.777889 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.778022 (+ 133us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:13.778280 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:13.779662 (+ 1382us) server_negotiation.cc:299] Negotiation successful
0806 09:25:13.779862 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2558,"thread_start_us":2336,"threads_started":1}
I20260806 09:25:13.780740 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:13.765115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42758 (local address 127.16.86.130:34417)
0806 09:25:13.768884 (+ 3769us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:13.768897 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:13.768931 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:25:13.769043 (+ 112us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:13.769051 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:13.769365 (+ 314us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:13.770978 (+ 1613us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:13.771328 (+ 350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.773007 (+ 1679us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.775916 (+ 2909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:13.777342 (+ 1426us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:13.779293 (+ 1951us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:13.779794 (+ 501us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:13.779886 (+ 92us) server_negotiation.cc:299] Negotiation successful
0806 09:25:13.780113 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3522,"thread_start_us":3355,"threads_started":1}
I20260806 09:25:13.783694 4493 tablet_service.cc:1467] Processing CreateTablet for tablet 88eb1c602cdf4a84b00ad622001ce596 (DEFAULT_TABLE table=test-table [id=782229f82d964d3a9fea0a85e8fe6f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:25:13.784590 4852 tablet_service.cc:1467] Processing CreateTablet for tablet 88eb1c602cdf4a84b00ad622001ce596 (DEFAULT_TABLE table=test-table [id=782229f82d964d3a9fea0a85e8fe6f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:25:13.784586 4674 tablet_service.cc:1467] Processing CreateTablet for tablet 88eb1c602cdf4a84b00ad622001ce596 (DEFAULT_TABLE table=test-table [id=782229f82d964d3a9fea0a85e8fe6f8e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:25:13.785668 4493 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88eb1c602cdf4a84b00ad622001ce596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:13.786206 4674 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88eb1c602cdf4a84b00ad622001ce596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:13.786202 4852 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 88eb1c602cdf4a84b00ad622001ce596. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:13.803067 4961 tablet_bootstrap.cc:492] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d: Bootstrap starting.
I20260806 09:25:13.807976 4961 tablet_bootstrap.cc:654] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:13.809401 4964 tablet_bootstrap.cc:492] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7: Bootstrap starting.
I20260806 09:25:13.809628 4961 log.cc:824] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:13.813612 4962 tablet_bootstrap.cc:492] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a: Bootstrap starting.
I20260806 09:25:13.816990 4964 tablet_bootstrap.cc:654] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:13.817474 4962 tablet_bootstrap.cc:654] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:13.818393 4961 tablet_bootstrap.cc:492] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d: No bootstrap required, opened a new log
I20260806 09:25:13.818468 4964 log.cc:824] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:13.818657 4962 log.cc:824] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:13.818840 4961 ts_tablet_manager.cc:1397] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d: Time spent bootstrapping tablet: real 0.016s user 0.007s sys 0.005s
I20260806 09:25:13.822261 4962 tablet_bootstrap.cc:492] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a: No bootstrap required, opened a new log
I20260806 09:25:13.822510 4962 ts_tablet_manager.cc:1397] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20260806 09:25:13.822782 4964 tablet_bootstrap.cc:492] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7: No bootstrap required, opened a new log
I20260806 09:25:13.823195 4964 ts_tablet_manager.cc:1397] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7: Time spent bootstrapping tablet: real 0.014s user 0.011s sys 0.000s
I20260806 09:25:13.830925 4962 raft_consensus.cc:348] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } } peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } }
I20260806 09:25:13.831434 4962 raft_consensus.cc:374] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:13.831537 4962 raft_consensus.cc:729] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633faae4693e45bcbe46babba1396b2a, State: Initialized, Role: FOLLOWER
I20260806 09:25:13.831429 4961 raft_consensus.cc:348] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } } peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } }
I20260806 09:25:13.832046 4961 raft_consensus.cc:374] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:13.832171 4961 raft_consensus.cc:729] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e3a0be0611487c8dc56ee0e0863f6d, State: Initialized, Role: FOLLOWER
I20260806 09:25:13.832347 4962 consensus_queue.cc:260] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } } peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } }
I20260806 09:25:13.832947 4961 consensus_queue.cc:260] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } } peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } }
I20260806 09:25:13.833853 4931 heartbeater.cc:503] Master 127.16.86.190:33769 was elected leader, sending a full tablet report...
I20260806 09:25:13.834549 4962 ts_tablet_manager.cc:1428] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a: Time spent starting tablet: real 0.012s user 0.014s sys 0.000s
I20260806 09:25:13.835850 4964 raft_consensus.cc:348] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } } peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } }
I20260806 09:25:13.836472 4964 raft_consensus.cc:374] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:13.836596 4961 ts_tablet_manager.cc:1428] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d: Time spent starting tablet: real 0.018s user 0.008s sys 0.010s
I20260806 09:25:13.836593 4964 raft_consensus.cc:729] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e16385638a4d5b9655f89d5877a4c7, State: Initialized, Role: FOLLOWER
I20260806 09:25:13.837497 4964 consensus_queue.cc:260] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } } peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } }
I20260806 09:25:13.839895 4759 heartbeater.cc:503] Master 127.16.86.190:33769 was elected leader, sending a full tablet report...
I20260806 09:25:13.840128 4964 ts_tablet_manager.cc:1428] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7: Time spent starting tablet: real 0.017s user 0.016s sys 0.000s
W20260806 09:25:13.876202 4582 tablet.cc:2369] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:25:13.918038 4935 tablet.cc:2369] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:25:13.997927 4762 tablet.cc:2369] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:25:14.041363 4971 raft_consensus.cc:482] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:25:14.041706 4971 raft_consensus.cc:504] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } } peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } }
I20260806 09:25:14.043762 4971 leader_election.cc:288] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 26e3a0be0611487c8dc56ee0e0863f6d (127.16.86.129:39089), 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417)
I20260806 09:25:14.051498 4938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:14.044419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:34417 (local address 127.16.86.131:40183)
0806 09:25:14.044681 (+ 262us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:14.044705 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:14.044869 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:14.045623 (+ 754us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:14.045631 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:14.045668 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:14.045919 (+ 251us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:14.045929 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.047505 (+ 1576us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.047510 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:14.049134 (+ 1624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:14.049151 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.050563 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.050570 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:14.050799 (+ 229us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:14.050828 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 09:25:14.050948 (+ 120us) client_negotiation.cc:240] Negotiation successful
0806 09:25:14.051143 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":84,"mutex_wait_us":1}
I20260806 09:25:14.051481 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:14.044419 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:40183 (local address 127.16.86.130:34417)
0806 09:25:14.045095 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:14.045108 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:14.045152 (+ 44us) server_negotiation.cc:407] Connection header received
0806 09:25:14.045288 (+ 136us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:14.045296 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:14.045405 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:14.045563 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:14.046085 (+ 522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.047314 (+ 1229us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.049380 (+ 2066us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.050359 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.050605 (+ 246us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:14.050706 (+ 101us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:14.050992 (+ 286us) server_negotiation.cc:299] Negotiation successful
0806 09:25:14.051178 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":430}
I20260806 09:25:14.052091 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:14.044621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:58587 (local address 127.16.86.129:39089)
0806 09:25:14.044926 (+ 305us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:14.044936 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:14.045744 (+ 808us) server_negotiation.cc:407] Connection header received
0806 09:25:14.045917 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:14.045921 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:14.046029 (+ 108us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:14.046154 (+ 125us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:14.046679 (+ 525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.047764 (+ 1085us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.049875 (+ 2111us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.050806 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.051082 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:14.051189 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:14.051572 (+ 383us) server_negotiation.cc:299] Negotiation successful
0806 09:25:14.051740 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":109}
I20260806 09:25:14.052281 4974 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:14.044642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39089 (local address 127.16.86.131:58587)
0806 09:25:14.045589 (+ 947us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:14.045612 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:14.045752 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:14.046176 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:14.046179 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:14.046198 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:14.046483 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:14.046489 (+ 6us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.048002 (+ 1513us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.048008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:14.049655 (+ 1647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:14.049674 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.051128 (+ 1454us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.051135 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:14.051359 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:14.051390 (+ 31us) client_negotiation.cc:769] Sending connection context
0806 09:25:14.051536 (+ 146us) client_negotiation.cc:240] Negotiation successful
0806 09:25:14.051711 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":789,"mutex_wait_us":5,"thread_start_us":549,"threads_started":1}
I20260806 09:25:14.053484 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88eb1c602cdf4a84b00ad622001ce596" candidate_uuid: "633faae4693e45bcbe46babba1396b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e16385638a4d5b9655f89d5877a4c7" is_pre_election: true
I20260806 09:25:14.053938 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88eb1c602cdf4a84b00ad622001ce596" candidate_uuid: "633faae4693e45bcbe46babba1396b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" is_pre_election: true
I20260806 09:25:14.054215 4701 raft_consensus.cc:2454] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633faae4693e45bcbe46babba1396b2a in term 0.
I20260806 09:25:14.054647 4526 raft_consensus.cc:2454] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 633faae4693e45bcbe46babba1396b2a in term 0.
I20260806 09:25:14.055511 4794 leader_election.cc:302] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e16385638a4d5b9655f89d5877a4c7, 633faae4693e45bcbe46babba1396b2a; no voters:
I20260806 09:25:14.056104 4971 raft_consensus.cc:2781] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:25:14.056322 4971 raft_consensus.cc:482] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:25:14.056461 4971 raft_consensus.cc:3037] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:14.058831 4971 raft_consensus.cc:504] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } } peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } }
I20260806 09:25:14.059619 4971 leader_election.cc:288] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [CANDIDATE]: Term 1 election: Requested vote from peers 26e3a0be0611487c8dc56ee0e0863f6d (127.16.86.129:39089), 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417)
I20260806 09:25:14.060128 4526 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88eb1c602cdf4a84b00ad622001ce596" candidate_uuid: "633faae4693e45bcbe46babba1396b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e3a0be0611487c8dc56ee0e0863f6d"
I20260806 09:25:14.060365 4526 raft_consensus.cc:3037] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:14.060454 4701 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "88eb1c602cdf4a84b00ad622001ce596" candidate_uuid: "633faae4693e45bcbe46babba1396b2a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "42e16385638a4d5b9655f89d5877a4c7"
I20260806 09:25:14.060776 4701 raft_consensus.cc:3037] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:14.062907 4526 raft_consensus.cc:2454] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633faae4693e45bcbe46babba1396b2a in term 1.
I20260806 09:25:14.063563 4795 leader_election.cc:302] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e3a0be0611487c8dc56ee0e0863f6d, 633faae4693e45bcbe46babba1396b2a; no voters:
I20260806 09:25:14.063802 4701 raft_consensus.cc:2454] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 633faae4693e45bcbe46babba1396b2a in term 1.
I20260806 09:25:14.064050 4971 raft_consensus.cc:2781] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:25:14.064760 4971 raft_consensus.cc:686] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 1 LEADER]: Becoming Leader. State: Replica: 633faae4693e45bcbe46babba1396b2a, State: Running, Role: LEADER
I20260806 09:25:14.065423 4971 consensus_queue.cc:237] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } } peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } }
I20260806 09:25:14.071667 4325 catalog_manager.cc:5495] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a reported cstate change: term changed from 0 to 1, leader changed from <none> to 633faae4693e45bcbe46babba1396b2a (127.16.86.131). New cstate: current_term: 1 leader_uuid: "633faae4693e45bcbe46babba1396b2a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } health_report { overall_health: HEALTHY } } }
I20260806 09:25:14.162508 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:14.155498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34904 (local address 127.16.86.131:38173)
0806 09:25:14.155850 (+ 352us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:14.155858 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:14.155888 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:25:14.156050 (+ 162us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:14.156057 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:14.156220 (+ 163us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:14.156424 (+ 204us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:25:14.156855 (+ 431us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.157825 (+ 970us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.159465 (+ 1640us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.160007 (+ 542us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.160179 (+ 172us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:14.160757 (+ 578us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:25:14.161571 (+ 814us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:25:14.161720 (+ 149us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:14.161936 (+ 216us) server_negotiation.cc:299] Negotiation successful
0806 09:25:14.162124 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":153}
I20260806 09:25:14.173130 4526 raft_consensus.cc:1261] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Refusing update from remote peer 633faae4693e45bcbe46babba1396b2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:25:14.173609 4701 raft_consensus.cc:1261] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7 [term 1 FOLLOWER]: Refusing update from remote peer 633faae4693e45bcbe46babba1396b2a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:25:14.174274 4975 consensus_queue.cc:1035] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:25:14.174820 4971 consensus_queue.cc:1035] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [LEADER]: Connected to new peer: Peer: permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:25:14.189976 4978 mvcc.cc:205] Tried to move back new op lower bound from 7315490054841151488 to 7315490054416281600. Current Snapshot: MvccSnapshot[applied={T|T < 7315490054841151488}]
I20260806 09:25:14.208078 4325 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:25:14.208433 4325 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:48410:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:25:14.213227 4325 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Sending DeleteTablet for 3 replicas of tablet 88eb1c602cdf4a84b00ad622001ce596
I20260806 09:25:14.214835 4674 tablet_service.cc:1514] Processing DeleteTablet for tablet 88eb1c602cdf4a84b00ad622001ce596 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:25:14 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:42758
I20260806 09:25:14.215067 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 4406
I20260806 09:25:14.214949 4493 tablet_service.cc:1514] Processing DeleteTablet for tablet 88eb1c602cdf4a84b00ad622001ce596 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:25:14 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:49432
I20260806 09:25:14.216969 4852 tablet_service.cc:1514] Processing DeleteTablet for tablet 88eb1c602cdf4a84b00ad622001ce596 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:25:14 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:34890
I20260806 09:25:14.225549 4993 tablet_replica.cc:330] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d: stopping tablet replica
I20260806 09:25:14.226126 4993 raft_consensus.cc:2229] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:25:14.226662 4996 tablet_replica.cc:330] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a: stopping tablet replica
I20260806 09:25:14.226856 4993 raft_consensus.cc:2258] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:25:14.227519 4996 raft_consensus.cc:2229] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:25:14.227983 4992 tablet_replica.cc:330] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7: stopping tablet replica
I20260806 09:25:14.228612 4992 raft_consensus.cc:2229] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:25:14.229175 4992 raft_consensus.cc:2258] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:25:14.229322 4996 raft_consensus.cc:2258] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:25:14.231410 4993 ts_tablet_manager.cc:1905] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:25:14.232511 4992 ts_tablet_manager.cc:1905] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:25:14.232511 4996 ts_tablet_manager.cc:1905] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:25:14.239033 4993 ts_tablet_manager.cc:1918] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:25:14.239255 4993 log.cc:1196] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/88eb1c602cdf4a84b00ad622001ce596
I20260806 09:25:14.239873 4993 ts_tablet_manager.cc:1939] T 88eb1c602cdf4a84b00ad622001ce596 P 26e3a0be0611487c8dc56ee0e0863f6d: Deleting consensus metadata
I20260806 09:25:14.242463 4303 catalog_manager.cc:4841] TS 26e3a0be0611487c8dc56ee0e0863f6d (127.16.86.129:39089): tablet 88eb1c602cdf4a84b00ad622001ce596 (table test-table [id=782229f82d964d3a9fea0a85e8fe6f8e]) successfully deleted
Aug 06 09:25:14 dist-test-slave-h3ft krb5kdc[4262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:25:14.244760 4996 ts_tablet_manager.cc:1918] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:25:14.245119 4996 log.cc:1196] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/88eb1c602cdf4a84b00ad622001ce596
I20260806 09:25:14.245155 4992 ts_tablet_manager.cc:1918] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:25:14.245380 4992 log.cc:1196] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/88eb1c602cdf4a84b00ad622001ce596
I20260806 09:25:14.246065 4992 ts_tablet_manager.cc:1939] T 88eb1c602cdf4a84b00ad622001ce596 P 42e16385638a4d5b9655f89d5877a4c7: Deleting consensus metadata
I20260806 09:25:14.246052 4996 ts_tablet_manager.cc:1939] T 88eb1c602cdf4a84b00ad622001ce596 P 633faae4693e45bcbe46babba1396b2a: Deleting consensus metadata
I20260806 09:25:14.248698 4302 catalog_manager.cc:4841] TS 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417): tablet 88eb1c602cdf4a84b00ad622001ce596 (table test-table [id=782229f82d964d3a9fea0a85e8fe6f8e]) successfully deleted
I20260806 09:25:14.248732 4301 catalog_manager.cc:4841] TS 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173): tablet 88eb1c602cdf4a84b00ad622001ce596 (table test-table [id=782229f82d964d3a9fea0a85e8fe6f8e]) successfully deleted
I20260806 09:25:14.252871 4958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:14.216353 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49438 (local address 127.16.86.129:39089)
0806 09:25:14.216575 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:14.216581 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:14.225763 (+ 9182us) server_negotiation.cc:407] Connection header received
0806 09:25:14.226020 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:14.226026 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:14.226139 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:14.226314 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:14.230394 (+ 4080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.231561 (+ 1167us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.233660 (+ 2099us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.234418 (+ 758us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.234530 (+ 112us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:14.245378 (+ 10848us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:14.245398 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:14.245424 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:14.245515 (+ 91us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:14.249595 (+ 4080us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:14.250464 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:14.250472 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:14.250475 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:14.250621 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:14.251080 (+ 459us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:14.251084 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:14.251086 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:14.251498 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:14.251847 (+ 349us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:14.252208 (+ 361us) server_negotiation.cc:299] Negotiation successful
0806 09:25:14.252463 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 09:25:14.253748 4576 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:14.360946 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 4406
I20260806 09:25:14.397238 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 4590
Aug 06 09:25:14 dist-test-slave-h3ft krb5kdc[4262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:25:14.418511 4959 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:14.398420 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42768 (local address 127.16.86.130:34417)
0806 09:25:14.398733 (+ 313us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:14.398740 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:14.398767 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:25:14.398972 (+ 205us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:14.398979 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:14.399136 (+ 157us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:14.399273 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:14.400995 (+ 1722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.401901 (+ 906us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.403781 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.404294 (+ 513us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.404471 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:14.410378 (+ 5907us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:14.410403 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:14.410427 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:14.410491 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:14.414572 (+ 4081us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:14.415788 (+ 1216us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:14.415798 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:14.415801 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:14.416014 (+ 213us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:14.416498 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:14.416502 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:14.416504 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:14.417008 (+ 504us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:14.417382 (+ 374us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:14.417790 (+ 408us) server_negotiation.cc:299] Negotiation successful
0806 09:25:14.418086 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":125}
I20260806 09:25:14.419648 4754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:14.524483 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 4590
I20260806 09:25:14.556739 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 4768
Aug 06 09:25:14 dist-test-slave-h3ft krb5kdc[4262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008310, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:25:14.577615 4960 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:14.557985 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34912 (local address 127.16.86.131:38173)
0806 09:25:14.558296 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:14.558304 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:14.558333 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:25:14.558561 (+ 228us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:14.558567 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:14.558679 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:14.558833 (+ 154us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:14.560733 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.561632 (+ 899us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.563239 (+ 1607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:14.563756 (+ 517us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:14.563955 (+ 199us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:14.569329 (+ 5374us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:14.569347 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:14.569370 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:14.569433 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:14.573166 (+ 3733us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:14.574147 (+ 981us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:14.574178 (+ 31us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:14.574181 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:14.574430 (+ 249us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:14.575352 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:14.575360 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:14.575364 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:14.576026 (+ 662us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:14.576436 (+ 410us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:14.576839 (+ 403us) server_negotiation.cc:299] Negotiation successful
0806 09:25:14.577159 (+ 320us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260806 09:25:14.578564 4924 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:14.708570 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 4768
I20260806 09:25:14.730901 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 4278
I20260806 09:25:14.732613 4364 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:14.855202 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 4278
I20260806 09:25:14.887686 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:33769
--webserver_interface=127.16.86.190
--webserver_port=33311
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:33769
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:25:15.342269 5003 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:15.342700 5003 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:15.342895 5003 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:15.351327 5003 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:25:15.351485 5003 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:15.351576 5003 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:15.351675 5003 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:25:15.351786 5003 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:25:15.363647 5003 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:33769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:33769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:15.365527 5003 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:15.367398 5003 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:15.401865 5015 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:15.401932 5011 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:15.401954 5010 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:15.403326 5003 server_base.cc:1034] running on GCE node
I20260806 09:25:15.404639 5003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:15.406948 5003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:15.408283 5003 hybrid_clock.cc:648] HybridClock initialized: now 1786008315408203 us; error 49 us; skew 500 ppm
I20260806 09:25:15.408895 5003 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:15 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:15.413862 5003 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:25:15.416564 5003 webserver.cc:460] Webserver started at http://127.16.86.190:33311/ using document root <none> and password file <none>
I20260806 09:25:15.417665 5003 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:15.417876 5003 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:15.426447 5003 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.001s sys 0.005s
I20260806 09:25:15.430984 5022 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:15.433202 5003 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260806 09:25:15.433525 5003 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "00198bb056544817a977c392a45892ab"
format_stamp: "Formatted at 2026-08-06 09:25:11 on dist-test-slave-h3ft"
server_key: "9438fd7507bb5ab45c90bd984cf9a708"
server_key_iv: "843373a7cb1769d9935403164bbb9b67"
server_key_version: "encryptionkey@0"
I20260806 09:25:15.434960 5003 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:15.449288 5003 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:15.454919 5003 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:15.455353 5003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:15.480198 5003 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:33769
I20260806 09:25:15.480263 5095 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:33769 every 8 connection(s)
I20260806 09:25:15.482239 5003 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:25:15.490424 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 5003
I20260806 09:25:15.491482 5096 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:25:15.491755 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:39089
--local_ip_for_outbound_sockets=127.16.86.129
--tserver_master_addrs=127.16.86.190:33769
--webserver_port=34609
--webserver_interface=127.16.86.129
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:25:15.497743 5096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Bootstrap starting.
I20260806 09:25:15.528141 5096 log.cc:824] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:15.548404 5096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:25:15.549202 5096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Bootstrap complete.
I20260806 09:25:15.564968 5096 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } }
I20260806 09:25:15.566401 5096 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00198bb056544817a977c392a45892ab, State: Initialized, Role: FOLLOWER
I20260806 09:25:15.567441 5096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } }
I20260806 09:25:15.567824 5096 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:25:15.568049 5096 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:25:15.568296 5096 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:25:15.575331 5096 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } }
I20260806 09:25:15.576207 5096 leader_election.cc:302] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00198bb056544817a977c392a45892ab; no voters:
I20260806 09:25:15.577040 5096 leader_election.cc:288] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 09:25:15.577363 5105 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:25:15.579928 5096 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:25:15.583480 5105 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 2 LEADER]: Becoming Leader. State: Replica: 00198bb056544817a977c392a45892ab, State: Running, Role: LEADER
I20260806 09:25:15.584709 5105 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } }
I20260806 09:25:15.589223 5106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "00198bb056544817a977c392a45892ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } } }
I20260806 09:25:15.590343 5106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:15.590916 5107 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00198bb056544817a977c392a45892ab. Latest consensus state: current_term: 2 leader_uuid: "00198bb056544817a977c392a45892ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } } }
I20260806 09:25:15.591910 5107 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:15.596704 5125 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:25:15.607043 5125 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:25:15.608753 5125 catalog_manager.cc:1259] Loaded cluster ID: e083fe5589f54e90b261b02584efec5f
I20260806 09:25:15.608950 5125 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:25:15.612960 5125 catalog_manager.cc:1314] IPKI private key was stored in cleartext, attempting to encrypt it.
I20260806 09:25:15.633385 5125 catalog_manager.cc:1324] IPKI private key successfully encrypted.
I20260806 09:25:15.645413 5125 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:25:15.647683 5125 catalog_manager.cc:5879] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Loaded TSK: 0
I20260806 09:25:15.649724 5125 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:25:15.937237 5099 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:15.937604 5099 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:15.937781 5099 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:15.947726 5099 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:15.947904 5099 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:15.948175 5099 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:25:15.958905 5099 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:39089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=34609
--tserver_master_addrs=127.16.86.190:33769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:15.960455 5099 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:15.961905 5099 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:15.979683 5137 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:15.980178 5146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:15.981065 5140 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:15.982781 5099 server_base.cc:1034] running on GCE node
I20260806 09:25:15.983896 5099 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:15.985641 5099 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:15.986964 5099 hybrid_clock.cc:648] HybridClock initialized: now 1786008315986862 us; error 87 us; skew 500 ppm
I20260806 09:25:15.987494 5099 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:15 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:15.992401 5099 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:25:15.994473 5099 webserver.cc:460] Webserver started at http://127.16.86.129:34609/ using document root <none> and password file <none>
I20260806 09:25:15.995467 5099 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:15.995653 5099 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:16.002378 5099 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:25:16.006011 5155 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:16.008029 5099 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:25:16.008318 5099 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "26e3a0be0611487c8dc56ee0e0863f6d"
format_stamp: "Formatted at 2026-08-06 09:25:12 on dist-test-slave-h3ft"
server_key: "85b70fa9fb01dd7214f5065609bdcc67"
server_key_iv: "c887b9883a683eb02be0d45df5c2d44a"
server_key_version: "encryptionkey@0"
I20260806 09:25:16.009483 5099 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:16.052628 5099 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:16.058004 5099 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:16.058393 5099 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:16.059748 5099 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:25:16.061462 5099 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:25:16.061616 5099 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:16.061776 5099 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:25:16.061882 5099 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:16.107671 5099 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:39089
I20260806 09:25:16.107813 5308 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:39089 every 8 connection(s)
I20260806 09:25:16.109511 5099 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:25:16.111595 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 5099
I20260806 09:25:16.112625 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:34417
--local_ip_for_outbound_sockets=127.16.86.130
--tserver_master_addrs=127.16.86.190:33769
--webserver_port=44299
--webserver_interface=127.16.86.130
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:25:16 dist-test-slave-h3ft krb5kdc[4262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008315, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:16.148033 5312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:16.114466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:60517)
0806 09:25:16.116334 (+ 1868us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:16.116416 (+ 82us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:16.119922 (+ 3506us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:16.121882 (+ 1960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:16.121899 (+ 17us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:16.122981 (+ 1082us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:16.124093 (+ 1112us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:16.124128 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:16.126159 (+ 2031us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:16.126168 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:16.128902 (+ 2734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:16.128924 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:16.130803 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:16.130811 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:16.131214 (+ 403us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:16.132307 (+ 1093us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:16.132346 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:16.135721 (+ 3375us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:16.141533 (+ 5812us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:16.141545 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:16.141566 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:16.142479 (+ 913us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:16.143230 (+ 751us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:16.143236 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:16.143239 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:16.143514 (+ 275us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:16.144761 (+ 1247us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:16.144771 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:16.145350 (+ 579us) client_negotiation.cc:769] Sending connection context
0806 09:25:16.145688 (+ 338us) client_negotiation.cc:240] Negotiation successful
0806 09:25:16.146313 (+ 625us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1444,"thread_start_us":1098,"threads_started":1}
I20260806 09:25:16.148375 5311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:16.114205 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:60517 (local address 127.16.86.190:33769)
0806 09:25:16.115440 (+ 1235us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:16.115467 (+ 27us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:16.116639 (+ 1172us) server_negotiation.cc:407] Connection header received
0806 09:25:16.120748 (+ 4109us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:16.120765 (+ 17us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:16.121174 (+ 409us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:16.121554 (+ 380us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:16.124429 (+ 2875us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:16.125905 (+ 1476us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:16.129137 (+ 3232us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:16.130436 (+ 1299us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:16.130712 (+ 276us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:16.136149 (+ 5437us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:16.136163 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:16.136191 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:16.136264 (+ 73us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:16.141246 (+ 4982us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:16.142727 (+ 1481us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:16.142740 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:16.142754 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:16.143052 (+ 298us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:16.143700 (+ 648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:16.143709 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:16.143715 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:16.144545 (+ 830us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:16.144808 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:16.146077 (+ 1269us) server_negotiation.cc:299] Negotiation successful
0806 09:25:16.146692 (+ 615us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":896,"thread_start_us":740,"threads_started":1}
I20260806 09:25:16.152031 5309 heartbeater.cc:347] Connected to a master server at 127.16.86.190:33769
I20260806 09:25:16.152474 5309 heartbeater.cc:464] Registering TS with master...
I20260806 09:25:16.153528 5309 heartbeater.cc:511] Master 127.16.86.190:33769 requested a full tablet report, sending...
I20260806 09:25:16.156477 5049 ts_manager.cc:194] Registered new tserver with Master: 26e3a0be0611487c8dc56ee0e0863f6d (127.16.86.129:39089)
I20260806 09:25:16.158352 5049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:60517
W20260806 09:25:16.526733 5313 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:16.527307 5313 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:16.527513 5313 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:16.536070 5313 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:16.536273 5313 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:16.536468 5313 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:25:16.548058 5313 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:34417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=44299
--tserver_master_addrs=127.16.86.190:33769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:16.549922 5313 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:16.551720 5313 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:16.568950 5324 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:16.568969 5320 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:16.569097 5326 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:16.570303 5313 server_base.cc:1034] running on GCE node
I20260806 09:25:16.571257 5313 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:16.572686 5313 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:16.573999 5313 hybrid_clock.cc:648] HybridClock initialized: now 1786008316573907 us; error 75 us; skew 500 ppm
I20260806 09:25:16.574455 5313 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:16 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:16.578534 5313 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:25:16.580674 5313 webserver.cc:460] Webserver started at http://127.16.86.130:44299/ using document root <none> and password file <none>
I20260806 09:25:16.581844 5313 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:16.582059 5313 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:16.589465 5313 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:25:16.593278 5335 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:16.595391 5313 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:25:16.595657 5313 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "42e16385638a4d5b9655f89d5877a4c7"
format_stamp: "Formatted at 2026-08-06 09:25:12 on dist-test-slave-h3ft"
server_key: "cf283098b82044fbb6a46ce61fa8ecca"
server_key_iv: "684ef4a1e5f7e50fdcd91b22f0e49dc6"
server_key_version: "encryptionkey@0"
I20260806 09:25:16.597127 5313 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:16.624193 5313 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:16.630151 5313 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:16.630472 5313 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:16.631799 5313 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:25:16.633631 5313 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:25:16.633740 5313 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:16.633836 5313 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:25:16.633896 5313 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:16.678696 5313 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:34417
I20260806 09:25:16.678845 5510 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:34417 every 8 connection(s)
I20260806 09:25:16.680727 5313 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:25:16.688498 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 5313
I20260806 09:25:16.689306 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:38173
--local_ip_for_outbound_sockets=127.16.86.131
--tserver_master_addrs=127.16.86.190:33769
--webserver_port=40153
--webserver_interface=127.16.86.131
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:25:16 dist-test-slave-h3ft krb5kdc[4262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008316, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:16.710372 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:16.684087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:57697 (local address 127.16.86.190:33769)
0806 09:25:16.684750 (+ 663us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:16.684755 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:16.686030 (+ 1275us) server_negotiation.cc:407] Connection header received
0806 09:25:16.688625 (+ 2595us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:16.688633 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:16.688760 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:16.688919 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:16.691252 (+ 2333us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:16.692211 (+ 959us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:16.695564 (+ 3353us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:16.696116 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:16.696283 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:16.702150 (+ 5867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:16.702156 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:16.702160 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:16.702257 (+ 97us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:16.705946 (+ 3689us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:16.707249 (+ 1303us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:16.707254 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:16.707257 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:16.707431 (+ 174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:16.707993 (+ 562us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:16.707998 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:16.708000 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:16.708371 (+ 371us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:16.708562 (+ 191us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:16.709536 (+ 974us) server_negotiation.cc:299] Negotiation successful
0806 09:25:16.709840 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":525,"thread_start_us":383,"threads_started":1}
I20260806 09:25:16.711902 5514 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:16.684621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:57697)
0806 09:25:16.685720 (+ 1099us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:16.685775 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:16.688229 (+ 2454us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:16.689267 (+ 1038us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:16.689279 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:16.690105 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:16.690913 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:16.690971 (+ 58us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:16.692525 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:16.692537 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:16.695274 (+ 2737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:16.695306 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:16.696461 (+ 1155us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:16.696472 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:16.696902 (+ 430us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:16.697969 (+ 1067us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:16.698019 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:16.701711 (+ 3692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:16.706173 (+ 4462us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:16.706186 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:16.706219 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:16.706995 (+ 776us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:16.707569 (+ 574us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:16.707577 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:16.707581 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:16.707840 (+ 259us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:16.708530 (+ 690us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:16.708540 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:16.709056 (+ 516us) client_negotiation.cc:769] Sending connection context
0806 09:25:16.709424 (+ 368us) client_negotiation.cc:240] Negotiation successful
0806 09:25:16.710200 (+ 776us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":715,"thread_start_us":427,"threads_started":1}
I20260806 09:25:16.714339 5511 heartbeater.cc:347] Connected to a master server at 127.16.86.190:33769
I20260806 09:25:16.714893 5511 heartbeater.cc:464] Registering TS with master...
I20260806 09:25:16.716202 5511 heartbeater.cc:511] Master 127.16.86.190:33769 requested a full tablet report, sending...
I20260806 09:25:16.718741 5049 ts_manager.cc:194] Registered new tserver with Master: 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417)
I20260806 09:25:16.720337 5049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:57697
W20260806 09:25:17.064779 5515 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:17.065097 5515 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:17.065197 5515 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:17.072715 5515 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:17.072856 5515 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:17.072975 5515 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:25:17.083258 5515 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:38173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=40153
--tserver_master_addrs=127.16.86.190:33769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:17.084749 5515 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:17.086153 5515 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:17.101434 5520 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:17.101642 5522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:17.102865 5524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:17.103499 5515 server_base.cc:1034] running on GCE node
I20260806 09:25:17.104266 5515 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:17.105381 5515 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:17.106834 5515 hybrid_clock.cc:648] HybridClock initialized: now 1786008317106690 us; error 93 us; skew 500 ppm
I20260806 09:25:17.107266 5515 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:17 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008317, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:17.111310 5515 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:25:17.113224 5515 webserver.cc:460] Webserver started at http://127.16.86.131:40153/ using document root <none> and password file <none>
I20260806 09:25:17.114081 5515 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:17.114264 5515 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:17.120651 5515 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:25:17.124166 5531 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:17.125892 5515 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:25:17.126106 5515 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "633faae4693e45bcbe46babba1396b2a"
format_stamp: "Formatted at 2026-08-06 09:25:13 on dist-test-slave-h3ft"
server_key: "a140f6feb89dfebe83412ba1df2547c3"
server_key_iv: "eca9f49c0ad008dfe4c2ddbaadf25095"
server_key_version: "encryptionkey@0"
I20260806 09:25:17.127269 5515 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:17.146461 5515 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:17.151165 5515 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:17.151464 5515 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:17.152699 5515 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:25:17.154217 5515 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:25:17.154350 5515 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:17.154515 5515 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:25:17.154636 5515 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:17.162801 5309 heartbeater.cc:503] Master 127.16.86.190:33769 was elected leader, sending a full tablet report...
I20260806 09:25:17.198093 5515 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:38173
I20260806 09:25:17.198230 5682 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:38173 every 8 connection(s)
I20260806 09:25:17.200004 5515 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:25:17.207280 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 5515
Aug 06 09:25:17 dist-test-slave-h3ft krb5kdc[4262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008317, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:17.226686 5513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.203462 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:58821 (local address 127.16.86.190:33769)
0806 09:25:17.203679 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:17.203683 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:17.205145 (+ 1462us) server_negotiation.cc:407] Connection header received
0806 09:25:17.207591 (+ 2446us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:17.207599 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:17.207729 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:17.207854 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:17.210214 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.211103 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.213121 (+ 2018us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.213840 (+ 719us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.213983 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.219155 (+ 5172us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:17.219161 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:17.219165 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:17.219223 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:17.222898 (+ 3675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:17.223889 (+ 991us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:17.223896 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:17.223900 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:17.224041 (+ 141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:17.224526 (+ 485us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:17.224534 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:17.224541 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:17.224971 (+ 430us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:17.225155 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:17.225955 (+ 800us) server_negotiation.cc:299] Negotiation successful
0806 09:25:17.226265 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":36,"server-negotiator.queue_time_us":69}
I20260806 09:25:17.226691 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.208455 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48416 (local address 127.16.86.190:33769)
0806 09:25:17.209483 (+ 1028us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:17.209487 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:17.209501 (+ 14us) server_negotiation.cc:407] Connection header received
0806 09:25:17.209571 (+ 70us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:17.209575 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:17.209642 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:17.209772 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:17.211479 (+ 1707us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.212367 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.214276 (+ 1909us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.215039 (+ 763us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.215225 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.218643 (+ 3418us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:17.218651 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:17.218656 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:17.218710 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:17.222686 (+ 3976us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:17.223839 (+ 1153us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:17.223848 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:17.223852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:17.224028 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:17.224503 (+ 475us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:17.224509 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:17.224515 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:17.225007 (+ 492us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:17.225338 (+ 331us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:17.225743 (+ 405us) server_negotiation.cc:299] Negotiation successful
0806 09:25:17.226097 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":24,"server-negotiator.queue_time_us":908,"thread_start_us":754,"threads_started":1}
I20260806 09:25:17.227746 5686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.203946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:58821)
0806 09:25:17.204843 (+ 897us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.204894 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:17.207208 (+ 2314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:17.208187 (+ 979us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:17.208203 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:17.209247 (+ 1044us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:17.210019 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:17.210038 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.211279 (+ 1241us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.211284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:17.212909 (+ 1625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:17.212924 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.214222 (+ 1298us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.214232 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:17.214679 (+ 447us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.215712 (+ 1033us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:17.215740 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:17.218771 (+ 3031us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:17.223143 (+ 4372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:17.223157 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:17.223186 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:17.223752 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:17.224166 (+ 414us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:17.224170 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:17.224174 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:17.224397 (+ 223us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:17.225118 (+ 721us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:17.225128 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:17.225542 (+ 414us) client_negotiation.cc:769] Sending connection context
0806 09:25:17.225835 (+ 293us) client_negotiation.cc:240] Negotiation successful
0806 09:25:17.226439 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":332,"threads_started":1}
I20260806 09:25:17.229696 5683 heartbeater.cc:347] Connected to a master server at 127.16.86.190:33769
I20260806 09:25:17.230168 5683 heartbeater.cc:464] Registering TS with master...
I20260806 09:25:17.231398 5683 heartbeater.cc:511] Master 127.16.86.190:33769 requested a full tablet report, sending...
I20260806 09:25:17.233316 5049 ts_manager.cc:194] Registered new tserver with Master: 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173)
I20260806 09:25:17.234120 5049 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:58821
I20260806 09:25:17.242863 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:25:17.251828 5690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.244025 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48422 (local address 127.16.86.190:33769)
0806 09:25:17.244409 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:17.244420 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:17.244988 (+ 568us) server_negotiation.cc:407] Connection header received
0806 09:25:17.245209 (+ 221us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:17.245215 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:17.245435 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:17.245617 (+ 182us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:25:17.246103 (+ 486us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.246981 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.248709 (+ 1728us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.249337 (+ 628us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.249478 (+ 141us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.250110 (+ 632us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:25:17.250889 (+ 779us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:25:17.251146 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:17.251271 (+ 125us) server_negotiation.cc:299] Negotiation successful
0806 09:25:17.251437 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 09:25:17.253029 5049 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:48422:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:25:17.260811 5049 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:25:17.294823 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.276997 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42774 (local address 127.16.86.130:34417)
0806 09:25:17.279455 (+ 2458us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:17.279468 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:17.279509 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:25:17.279628 (+ 119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:17.279637 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:17.280132 (+ 495us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:17.280355 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:17.281932 (+ 1577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.284061 (+ 2129us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.288244 (+ 4183us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.290435 (+ 2191us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.290612 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.290908 (+ 296us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:17.294203 (+ 3295us) server_negotiation.cc:299] Negotiation successful
0806 09:25:17.294417 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":774,"thread_start_us":630,"threads_started":1}
I20260806 09:25:17.295432 5697 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.276784 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:34417 (local address 127.0.0.1:42774)
0806 09:25:17.278611 (+ 1827us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.278679 (+ 68us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:17.278948 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:17.280504 (+ 1556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:17.280512 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:17.280751 (+ 239us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:17.281375 (+ 624us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:17.281394 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.284488 (+ 3094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.284496 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:17.287868 (+ 3372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:17.287887 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.292670 (+ 4783us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.292677 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:17.293493 (+ 816us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.293574 (+ 81us) client_negotiation.cc:769] Sending connection context
0806 09:25:17.293719 (+ 145us) client_negotiation.cc:240] Negotiation successful
0806 09:25:17.293971 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1459,"thread_start_us":1251,"threads_started":1}
I20260806 09:25:17.300086 5400 tablet_service.cc:1467] Processing CreateTablet for tablet ee2eadeeacc74c5aa3cffc2c26f85cd4 (DEFAULT_TABLE table=test-table [id=57d3d91e2e4940fcb298ff7c96722e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:25:17.301827 5699 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.277189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:38173 (local address 127.0.0.1:34916)
0806 09:25:17.291674 (+ 14485us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.291702 (+ 28us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:17.291859 (+ 157us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:17.293746 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:17.293750 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:17.293778 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:17.294342 (+ 564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:17.294356 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.296582 (+ 2226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.296589 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:17.298497 (+ 1908us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:17.298512 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.300839 (+ 2327us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.300845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:17.301006 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.301047 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:25:17.301151 (+ 104us) client_negotiation.cc:240] Negotiation successful
0806 09:25:17.301320 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14267,"thread_start_us":14041,"threads_started":1}
I20260806 09:25:17.302006 5400 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee2eadeeacc74c5aa3cffc2c26f85cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:17.301805 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.278165 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34916 (local address 127.16.86.131:38173)
0806 09:25:17.292564 (+ 14399us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:17.292576 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:17.292614 (+ 38us) server_negotiation.cc:407] Connection header received
0806 09:25:17.292732 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:17.292741 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:17.293209 (+ 468us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:17.293456 (+ 247us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:17.294612 (+ 1156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.296405 (+ 1793us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.298720 (+ 2315us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.300324 (+ 1604us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.300470 (+ 146us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.300762 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:17.301232 (+ 470us) server_negotiation.cc:299] Negotiation successful
0806 09:25:17.301421 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14186,"thread_start_us":14008,"threads_started":1}
I20260806 09:25:17.303226 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.280491 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39089 (local address 127.0.0.1:49440)
0806 09:25:17.293199 (+ 12708us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.293220 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:17.293448 (+ 228us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:17.294721 (+ 1273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:17.294727 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:17.294754 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:17.295480 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:17.295492 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.297594 (+ 2102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.297599 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:17.299934 (+ 2335us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:17.299953 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.302255 (+ 2302us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.302262 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:17.302453 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.302492 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 09:25:17.302574 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 09:25:17.302762 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12539,"mutex_wait_us":3,"thread_start_us":12354,"threads_started":1}
I20260806 09:25:17.304497 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.281514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49440 (local address 127.16.86.129:39089)
0806 09:25:17.292536 (+ 11022us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:17.292570 (+ 34us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:17.293628 (+ 1058us) server_negotiation.cc:407] Connection header received
0806 09:25:17.293778 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:17.293790 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:17.294131 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:17.294394 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:17.295759 (+ 1365us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.297398 (+ 1639us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.300227 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.301912 (+ 1685us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.303632 (+ 1720us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.303895 (+ 263us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:17.303983 (+ 88us) server_negotiation.cc:299] Negotiation successful
0806 09:25:17.304169 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10758,"thread_start_us":10519,"threads_started":1}
I20260806 09:25:17.307040 5603 tablet_service.cc:1467] Processing CreateTablet for tablet ee2eadeeacc74c5aa3cffc2c26f85cd4 (DEFAULT_TABLE table=test-table [id=57d3d91e2e4940fcb298ff7c96722e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:25:17.308526 5223 tablet_service.cc:1467] Processing CreateTablet for tablet ee2eadeeacc74c5aa3cffc2c26f85cd4 (DEFAULT_TABLE table=test-table [id=57d3d91e2e4940fcb298ff7c96722e31]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:25:17.309585 5603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee2eadeeacc74c5aa3cffc2c26f85cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:17.309978 5223 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet ee2eadeeacc74c5aa3cffc2c26f85cd4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:17.316565 5704 tablet_bootstrap.cc:492] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7: Bootstrap starting.
I20260806 09:25:17.320214 5704 tablet_bootstrap.cc:654] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:17.321333 5704 log.cc:824] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:17.324218 5704 tablet_bootstrap.cc:492] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7: No bootstrap required, opened a new log
I20260806 09:25:17.324532 5704 ts_tablet_manager.cc:1397] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.000s
I20260806 09:25:17.328939 5706 tablet_bootstrap.cc:492] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: Bootstrap starting.
I20260806 09:25:17.329972 5709 tablet_bootstrap.cc:492] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a: Bootstrap starting.
I20260806 09:25:17.333295 5706 tablet_bootstrap.cc:654] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:17.334703 5706 log.cc:824] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:17.335115 5704 raft_consensus.cc:348] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:17.335688 5704 raft_consensus.cc:374] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:17.335753 5709 tablet_bootstrap.cc:654] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:17.335851 5704 raft_consensus.cc:729] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 42e16385638a4d5b9655f89d5877a4c7, State: Initialized, Role: FOLLOWER
I20260806 09:25:17.336594 5704 consensus_queue.cc:260] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:17.337643 5709 log.cc:824] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:17.339782 5511 heartbeater.cc:503] Master 127.16.86.190:33769 was elected leader, sending a full tablet report...
I20260806 09:25:17.340004 5706 tablet_bootstrap.cc:492] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: No bootstrap required, opened a new log
I20260806 09:25:17.340370 5706 ts_tablet_manager.cc:1397] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: Time spent bootstrapping tablet: real 0.012s user 0.009s sys 0.000s
I20260806 09:25:17.342561 5704 ts_tablet_manager.cc:1428] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7: Time spent starting tablet: real 0.018s user 0.011s sys 0.004s
I20260806 09:25:17.344008 5709 tablet_bootstrap.cc:492] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a: No bootstrap required, opened a new log
I20260806 09:25:17.344332 5709 ts_tablet_manager.cc:1397] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a: Time spent bootstrapping tablet: real 0.015s user 0.011s sys 0.000s
I20260806 09:25:17.352098 5706 raft_consensus.cc:348] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:17.352651 5706 raft_consensus.cc:374] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:17.352778 5706 raft_consensus.cc:729] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e3a0be0611487c8dc56ee0e0863f6d, State: Initialized, Role: FOLLOWER
I20260806 09:25:17.353533 5706 consensus_queue.cc:260] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:17.355378 5709 raft_consensus.cc:348] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:17.355836 5706 ts_tablet_manager.cc:1428] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20260806 09:25:17.355882 5709 raft_consensus.cc:374] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:17.355988 5709 raft_consensus.cc:729] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 633faae4693e45bcbe46babba1396b2a, State: Initialized, Role: FOLLOWER
I20260806 09:25:17.356619 5709 consensus_queue.cc:260] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:17.358237 5683 heartbeater.cc:503] Master 127.16.86.190:33769 was elected leader, sending a full tablet report...
I20260806 09:25:17.358754 5709 ts_tablet_manager.cc:1428] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a: Time spent starting tablet: real 0.014s user 0.014s sys 0.000s
W20260806 09:25:17.362411 5310 tablet.cc:2369] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:25:17.428308 5712 raft_consensus.cc:482] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:25:17.428726 5712 raft_consensus.cc:504] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:17.431288 5712 leader_election.cc:288] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173), 26e3a0be0611487c8dc56ee0e0863f6d (127.16.86.129:39089)
W20260806 09:25:17.433435 5512 tablet.cc:2369] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:25:17.440488 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.431936 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:36997 (local address 127.16.86.131:38173)
0806 09:25:17.432197 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:17.432205 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:17.433325 (+ 1120us) server_negotiation.cc:407] Connection header received
0806 09:25:17.433873 (+ 548us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:17.433879 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:17.433997 (+ 118us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:17.434072 (+ 75us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:17.435278 (+ 1206us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.436183 (+ 905us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.438394 (+ 2211us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.439308 (+ 914us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.439452 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.439512 (+ 60us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:17.440016 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 09:25:17.440174 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20260806 09:25:17.440652 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.431940 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:40029 (local address 127.16.86.129:39089)
0806 09:25:17.432200 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:17.432208 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:17.433634 (+ 1426us) server_negotiation.cc:407] Connection header received
0806 09:25:17.433896 (+ 262us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:17.433903 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:17.433973 (+ 70us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:17.434079 (+ 106us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:17.434876 (+ 797us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.435959 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.438289 (+ 2330us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.439441 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.439649 (+ 208us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.439749 (+ 100us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:17.440190 (+ 441us) server_negotiation.cc:299] Negotiation successful
0806 09:25:17.440347 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 09:25:17.440871 5717 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.432006 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39089 (local address 127.16.86.130:40029)
0806 09:25:17.433497 (+ 1491us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.433516 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:17.433733 (+ 217us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:17.434105 (+ 372us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:17.434110 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:17.434136 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:17.434577 (+ 441us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:17.434588 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.436133 (+ 1545us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.436138 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:17.438065 (+ 1927us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:17.438080 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.439733 (+ 1653us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.439738 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:17.439864 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.439893 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 09:25:17.440157 (+ 264us) client_negotiation.cc:240] Negotiation successful
0806 09:25:17.440355 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1300,"mutex_wait_us":116,"thread_start_us":1022,"threads_started":1}
I20260806 09:25:17.441299 5716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.431638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:38173 (local address 127.16.86.130:36997)
0806 09:25:17.433148 (+ 1510us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.433185 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:17.433502 (+ 317us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:17.434325 (+ 823us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:17.434329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:17.434368 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:17.435054 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:17.435076 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.436392 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.436397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:17.438108 (+ 1711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:17.438122 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.439570 (+ 1448us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.439577 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:17.439807 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.439849 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 09:25:17.439939 (+ 90us) client_negotiation.cc:240] Negotiation successful
0806 09:25:17.440243 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1286,"mutex_wait_us":2,"thread_start_us":886,"threads_started":1}
I20260806 09:25:17.442173 5251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2eadeeacc74c5aa3cffc2c26f85cd4" candidate_uuid: "42e16385638a4d5b9655f89d5877a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" is_pre_election: true
I20260806 09:25:17.442384 5630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2eadeeacc74c5aa3cffc2c26f85cd4" candidate_uuid: "42e16385638a4d5b9655f89d5877a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633faae4693e45bcbe46babba1396b2a" is_pre_election: true
I20260806 09:25:17.442904 5251 raft_consensus.cc:2454] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e16385638a4d5b9655f89d5877a4c7 in term 0.
I20260806 09:25:17.443059 5630 raft_consensus.cc:2454] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 42e16385638a4d5b9655f89d5877a4c7 in term 0.
I20260806 09:25:17.443888 5337 leader_election.cc:302] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 42e16385638a4d5b9655f89d5877a4c7, 633faae4693e45bcbe46babba1396b2a; no voters:
I20260806 09:25:17.444360 5712 raft_consensus.cc:2781] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:25:17.444545 5712 raft_consensus.cc:482] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:25:17.444677 5712 raft_consensus.cc:3037] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:17.447749 5712 raft_consensus.cc:504] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:17.448637 5712 leader_election.cc:288] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [CANDIDATE]: Term 1 election: Requested vote from peers 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173), 26e3a0be0611487c8dc56ee0e0863f6d (127.16.86.129:39089)
I20260806 09:25:17.449517 5630 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2eadeeacc74c5aa3cffc2c26f85cd4" candidate_uuid: "42e16385638a4d5b9655f89d5877a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "633faae4693e45bcbe46babba1396b2a"
I20260806 09:25:17.449684 5251 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "ee2eadeeacc74c5aa3cffc2c26f85cd4" candidate_uuid: "42e16385638a4d5b9655f89d5877a4c7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "26e3a0be0611487c8dc56ee0e0863f6d"
I20260806 09:25:17.449849 5630 raft_consensus.cc:3037] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:17.450059 5251 raft_consensus.cc:3037] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 0 FOLLOWER]: Advancing to term 1
W20260806 09:25:17.451970 5684 tablet.cc:2369] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:25:17.452813 5251 raft_consensus.cc:2454] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e16385638a4d5b9655f89d5877a4c7 in term 1.
I20260806 09:25:17.453504 5341 leader_election.cc:302] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 26e3a0be0611487c8dc56ee0e0863f6d, 42e16385638a4d5b9655f89d5877a4c7; no voters:
I20260806 09:25:17.453917 5712 raft_consensus.cc:2781] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:25:17.453964 5630 raft_consensus.cc:2454] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 42e16385638a4d5b9655f89d5877a4c7 in term 1.
I20260806 09:25:17.454607 5712 raft_consensus.cc:686] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 1 LEADER]: Becoming Leader. State: Replica: 42e16385638a4d5b9655f89d5877a4c7, State: Running, Role: LEADER
I20260806 09:25:17.455292 5712 consensus_queue.cc:237] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:17.460639 5049 catalog_manager.cc:5495] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 reported cstate change: term changed from 0 to 1, leader changed from <none> to 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130). New cstate: current_term: 1 leader_uuid: "42e16385638a4d5b9655f89d5877a4c7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } health_report { overall_health: UNKNOWN } } }
I20260806 09:25:17.490722 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.483262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42780 (local address 127.16.86.130:34417)
0806 09:25:17.483527 (+ 265us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:17.483532 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:17.483554 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:25:17.483833 (+ 279us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:17.483837 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:17.483953 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:17.484115 (+ 162us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:25:17.484644 (+ 529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.485466 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.487678 (+ 2212us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.488374 (+ 696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.488518 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.489221 (+ 703us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:25:17.489929 (+ 708us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:25:17.490079 (+ 150us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:17.490242 (+ 163us) server_negotiation.cc:299] Negotiation successful
0806 09:25:17.490330 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:25:17.502552 5251 raft_consensus.cc:1261] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Refusing update from remote peer 42e16385638a4d5b9655f89d5877a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:25:17.502701 5630 raft_consensus.cc:1261] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a [term 1 FOLLOWER]: Refusing update from remote peer 42e16385638a4d5b9655f89d5877a4c7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:25:17.503988 5719 consensus_queue.cc:1035] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:25:17.504521 5712 consensus_queue.cc:1035] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:25:17.519970 5726 mvcc.cc:205] Tried to move back new op lower bound from 7315490068475650048 to 7315490068301320192. Current Snapshot: MvccSnapshot[applied={T|T < 7315490068475650048}]
I20260806 09:25:17.536988 5047 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:48422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:25:17.537457 5047 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:48422:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:25:17.542519 5047 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Sending DeleteTablet for 3 replicas of tablet ee2eadeeacc74c5aa3cffc2c26f85cd4
I20260806 09:25:17.544337 5223 tablet_service.cc:1514] Processing DeleteTablet for tablet ee2eadeeacc74c5aa3cffc2c26f85cd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:25:17 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:49440
I20260806 09:25:17.544530 5603 tablet_service.cc:1514] Processing DeleteTablet for tablet ee2eadeeacc74c5aa3cffc2c26f85cd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:25:17 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:34916
I20260806 09:25:17.544530 5400 tablet_service.cc:1514] Processing DeleteTablet for tablet ee2eadeeacc74c5aa3cffc2c26f85cd4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:25:17 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:42774
I20260806 09:25:17.546694 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 5099
I20260806 09:25:17.565693 5742 tablet_replica.cc:330] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7: stopping tablet replica
I20260806 09:25:17.566825 5742 raft_consensus.cc:2229] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:25:17.568192 5742 raft_consensus.cc:2258] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:25:17.569154 5743 tablet_replica.cc:330] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a: stopping tablet replica
I20260806 09:25:17.570782 5703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.547808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49450 (local address 127.16.86.129:39089)
0806 09:25:17.548117 (+ 309us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:17.548127 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:17.548162 (+ 35us) server_negotiation.cc:407] Connection header received
0806 09:25:17.548276 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:17.548286 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:17.548417 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:17.548607 (+ 190us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:17.550608 (+ 2001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.552395 (+ 1787us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.554539 (+ 2144us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.555649 (+ 1110us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.555858 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.559763 (+ 3905us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:17.559795 (+ 32us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:17.559835 (+ 40us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:17.559921 (+ 86us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:17.565479 (+ 5558us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:17.566705 (+ 1226us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:17.566717 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:17.566721 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:17.566936 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:17.567477 (+ 541us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:17.567485 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:17.567490 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:17.568160 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:17.568483 (+ 323us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:17.570029 (+ 1546us) server_negotiation.cc:299] Negotiation successful
0806 09:25:17.570300 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 09:25:17.571610 5304 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:17.572494 5743 raft_consensus.cc:2229] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:25:17.574779 5742 ts_tablet_manager.cc:1905] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:25:17.575886 5743 raft_consensus.cc:2258] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:25:17.582543 5743 ts_tablet_manager.cc:1905] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:25:17.590104 5742 ts_tablet_manager.cc:1918] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:25:17.590387 5742 log.cc:1196] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/ee2eadeeacc74c5aa3cffc2c26f85cd4
I20260806 09:25:17.591126 5742 ts_tablet_manager.cc:1939] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 42e16385638a4d5b9655f89d5877a4c7: Deleting consensus metadata
I20260806 09:25:17.591663 5743 ts_tablet_manager.cc:1918] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:25:17.591964 5743 log.cc:1196] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/ee2eadeeacc74c5aa3cffc2c26f85cd4
I20260806 09:25:17.592692 5743 ts_tablet_manager.cc:1939] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 633faae4693e45bcbe46babba1396b2a: Deleting consensus metadata
I20260806 09:25:17.593756 5024 catalog_manager.cc:4841] TS 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417): tablet ee2eadeeacc74c5aa3cffc2c26f85cd4 (table test-table [id=57d3d91e2e4940fcb298ff7c96722e31]) successfully deleted
I20260806 09:25:17.595178 5023 catalog_manager.cc:4841] TS 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173): tablet ee2eadeeacc74c5aa3cffc2c26f85cd4 (table test-table [id=57d3d91e2e4940fcb298ff7c96722e31]) successfully deleted
I20260806 09:25:17.706010 5741 tablet_replica.cc:330] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: stopping tablet replica
I20260806 09:25:17.706614 5741 raft_consensus.cc:2229] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:25:17.707326 5741 raft_consensus.cc:2258] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:25:17.708530 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 5099
W20260806 09:25:17.747110 5025 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.86.129:39089 (error 108)
I20260806 09:25:17.747752 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 5313
I20260806 09:25:17.749840 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.749236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39089 (local address 127.0.0.1:49464)
0806 09:25:17.749492 (+ 256us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.749646 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260806 09:25:17.750237 5025 catalog_manager.cc:4584] TS 26e3a0be0611487c8dc56ee0e0863f6d (127.16.86.129:39089): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet ee2eadeeacc74c5aa3cffc2c26f85cd4: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39089: connect: Connection refused (error 111)
I20260806 09:25:17.766171 5698 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.748990 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42792 (local address 127.16.86.130:34417)
0806 09:25:17.749242 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:17.749249 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:17.749307 (+ 58us) server_negotiation.cc:407] Connection header received
0806 09:25:17.749404 (+ 97us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:17.749413 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:17.749516 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:17.749671 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:17.751744 (+ 2073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.752691 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.754320 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.754847 (+ 527us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.755013 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.758364 (+ 3351us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:17.758389 (+ 25us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:17.758411 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:17.758475 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:17.762514 (+ 4039us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:17.763491 (+ 977us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:17.763501 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:17.763503 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:17.763679 (+ 176us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:17.764135 (+ 456us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:17.764140 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:17.764142 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:17.764734 (+ 592us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:17.765167 (+ 433us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:17.765472 (+ 305us) server_negotiation.cc:299] Negotiation successful
0806 09:25:17.765736 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 09:25:17.767184 5502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:17.804144 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.803526 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39089 (local address 127.0.0.1:49474)
0806 09:25:17.803801 (+ 275us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.803951 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 09:25:17.805563 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.805065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39089 (local address 127.0.0.1:49488)
0806 09:25:17.805269 (+ 204us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.805397 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20260806 09:25:17.866688 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.866055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39089 (local address 127.0.0.1:49492)
0806 09:25:17.866350 (+ 295us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.866453 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 09:25:17.868078 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.867683 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39089 (local address 127.0.0.1:49502)
0806 09:25:17.867865 (+ 182us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.867957 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 09:25:17.891763 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 5313
I20260806 09:25:17.923328 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 5515
I20260806 09:25:17.941417 5701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.924421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34932 (local address 127.16.86.131:38173)
0806 09:25:17.924675 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:17.924680 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:17.924703 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:25:17.924834 (+ 131us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:17.924839 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:17.924965 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:17.925091 (+ 126us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:17.926622 (+ 1531us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.927590 (+ 968us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.929118 (+ 1528us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:17.929613 (+ 495us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:17.929751 (+ 138us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:17.933155 (+ 3404us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:17.933181 (+ 26us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:17.933207 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:17.933267 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:17.937383 (+ 4116us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:17.938400 (+ 1017us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:17.938410 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:17.938414 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:17.938614 (+ 200us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:17.939145 (+ 531us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:17.939151 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:17.939153 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:17.939823 (+ 670us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:17.940152 (+ 329us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:17.940576 (+ 424us) server_negotiation.cc:299] Negotiation successful
0806 09:25:17.940977 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":101}
I20260806 09:25:17.942337 5678 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:17.949407 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.948578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39089 (local address 127.0.0.1:49508)
0806 09:25:17.948956 (+ 378us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.949131 (+ 175us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20260806 09:25:17.951186 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:17.950496 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39089 (local address 127.0.0.1:49518)
0806 09:25:17.950803 (+ 307us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:17.950905 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":177}
I20260806 09:25:18.038741 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 5515
I20260806 09:25:18.068540 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 5003
I20260806 09:25:18.070588 5090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:18.190478 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 5003
I20260806 09:25:18.190579 5702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:18.189715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:39089 (local address 127.0.0.1:49532)
0806 09:25:18.190083 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:18.190281 (+ 198us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:39089: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20260806 09:25:18.226261 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:33769
--webserver_interface=127.16.86.190
--webserver_port=33311
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:33769
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:25:18.644757 5758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:18.645130 5758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:18.645231 5758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:18.652968 5758 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:25:18.653121 5758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:18.653169 5758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:18.653223 5758 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:25:18.653262 5758 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:25:18.663786 5758 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:33769
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:33769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=33311
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:18.665302 5758 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:18.666787 5758 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:18.681464 5764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:18.681519 5767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:18.681409 5765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:18.682775 5758 server_base.cc:1034] running on GCE node
I20260806 09:25:18.683779 5758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:18.685477 5758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:18.686765 5758 hybrid_clock.cc:648] HybridClock initialized: now 1786008318686654 us; error 81 us; skew 500 ppm
I20260806 09:25:18.687240 5758 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:18 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008318, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:18.690915 5758 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:25:18.693049 5758 webserver.cc:460] Webserver started at http://127.16.86.190:33311/ using document root <none> and password file <none>
I20260806 09:25:18.693810 5758 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:18.693930 5758 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:18.700466 5758 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:25:18.704573 5776 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:18.706629 5758 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:25:18.706835 5758 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "00198bb056544817a977c392a45892ab"
format_stamp: "Formatted at 2026-08-06 09:25:11 on dist-test-slave-h3ft"
server_key: "9438fd7507bb5ab45c90bd984cf9a708"
server_key_iv: "843373a7cb1769d9935403164bbb9b67"
server_key_version: "encryptionkey@0"
I20260806 09:25:18.708073 5758 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:18.726330 5758 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:18.731696 5758 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:18.731972 5758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:18.757316 5758 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:33769
I20260806 09:25:18.757416 5841 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:33769 every 8 connection(s)
I20260806 09:25:18.759253 5758 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:25:18.767843 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 5758
I20260806 09:25:18.768751 5842 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:25:18.769202 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:39089
--local_ip_for_outbound_sockets=127.16.86.129
--tserver_master_addrs=127.16.86.190:33769
--webserver_port=34609
--webserver_interface=127.16.86.129
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:25:18.775123 5842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Bootstrap starting.
I20260806 09:25:18.804306 5842 log.cc:824] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:18.824867 5842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:25:18.825613 5842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab: Bootstrap complete.
I20260806 09:25:18.837023 5842 raft_consensus.cc:348] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } }
I20260806 09:25:18.838195 5842 raft_consensus.cc:729] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 00198bb056544817a977c392a45892ab, State: Initialized, Role: FOLLOWER
I20260806 09:25:18.839334 5842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } }
I20260806 09:25:18.839690 5842 raft_consensus.cc:388] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:25:18.839843 5842 raft_consensus.cc:482] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:25:18.840057 5842 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 2 FOLLOWER]: Advancing to term 3
I20260806 09:25:18.847700 5842 raft_consensus.cc:504] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } }
I20260806 09:25:18.848601 5842 leader_election.cc:302] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 00198bb056544817a977c392a45892ab; no voters:
I20260806 09:25:18.849509 5842 leader_election.cc:288] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 09:25:18.850003 5846 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 3 FOLLOWER]: Leader election won for term 3
I20260806 09:25:18.852720 5842 sys_catalog.cc:564] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:25:18.857851 5846 raft_consensus.cc:686] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [term 3 LEADER]: Becoming Leader. State: Replica: 00198bb056544817a977c392a45892ab, State: Running, Role: LEADER
I20260806 09:25:18.858989 5846 consensus_queue.cc:237] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } }
I20260806 09:25:18.863543 5847 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "00198bb056544817a977c392a45892ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } } }
I20260806 09:25:18.864182 5847 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:18.865023 5852 sys_catalog.cc:455] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 00198bb056544817a977c392a45892ab. Latest consensus state: current_term: 3 leader_uuid: "00198bb056544817a977c392a45892ab" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "00198bb056544817a977c392a45892ab" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 33769 } } }
I20260806 09:25:18.865427 5852 sys_catalog.cc:458] T 00000000000000000000000000000000 P 00198bb056544817a977c392a45892ab [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:18.871491 5860 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:25:18.886189 5860 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:25:18.887908 5860 catalog_manager.cc:1259] Loaded cluster ID: e083fe5589f54e90b261b02584efec5f
I20260806 09:25:18.888092 5860 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
F20260806 09:25:18.890501 5860 catalog_manager.cc:1466] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7f01150ef9ad google::LogMessage::Fail() at ??:0
@ 0x7f01150f288a google::LogMessage::SendToLog() at ??:0
@ 0x7f01150ef4de google::LogMessage::Flush() at ??:0
@ 0x7f01150f0a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7f012ab26512 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7f012ab23976 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7f012abab610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7f012abab439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0117d3b345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7f0117d3e91c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7f0117d3e799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7f0117d19eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7f011aff36db start_thread at ??:0
@ 0x7f0113f8961f clone at ??:0
W20260806 09:25:19.290553 5844 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:19.291028 5844 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:19.291173 5844 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:19.299695 5844 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:19.299866 5844 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:19.300014 5844 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:25:19.311755 5844 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:39089
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=34609
--tserver_master_addrs=127.16.86.190:33769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:19.313592 5844 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:19.315351 5844 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:19.332373 5901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:19.332373 5906 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:19.332777 5902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:19.333984 5844 server_base.cc:1034] running on GCE node
I20260806 09:25:19.334841 5844 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:19.336200 5844 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:19.337457 5844 hybrid_clock.cc:648] HybridClock initialized: now 1786008319337345 us; error 100 us; skew 500 ppm
I20260806 09:25:19.337944 5844 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:19 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008319, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:19.347895 5844 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:25:19.350188 5844 webserver.cc:460] Webserver started at http://127.16.86.129:34609/ using document root <none> and password file <none>
I20260806 09:25:19.351261 5844 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:19.351457 5844 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:19.358963 5844 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:25:19.363462 5913 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:19.365710 5844 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.000s sys 0.003s
I20260806 09:25:19.366060 5844 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "26e3a0be0611487c8dc56ee0e0863f6d"
format_stamp: "Formatted at 2026-08-06 09:25:12 on dist-test-slave-h3ft"
server_key: "85b70fa9fb01dd7214f5065609bdcc67"
server_key_iv: "c887b9883a683eb02be0d45df5c2d44a"
server_key_version: "encryptionkey@0"
I20260806 09:25:19.367671 5844 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:19.391978 5844 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:19.398769 5844 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:19.399183 5844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:19.400591 5844 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:25:19.403209 5922 ts_tablet_manager.cc:542] Loading tablet metadata (0/1 complete)
I20260806 09:25:19.410176 5844 ts_tablet_manager.cc:579] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20260806 09:25:19.410449 5844 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.008s user 0.001s sys 0.000s
I20260806 09:25:19.410655 5844 ts_tablet_manager.cc:594] Registering tablets (0/1 complete)
I20260806 09:25:19.415184 5844 ts_tablet_manager.cc:610] Registered 1 tablets
I20260806 09:25:19.415349 5844 ts_tablet_manager.cc:589] Time spent register tablets: real 0.005s user 0.005s sys 0.000s
I20260806 09:25:19.416114 5922 tablet_bootstrap.cc:492] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: Bootstrap starting.
I20260806 09:25:19.447310 5922 log.cc:824] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:19.465014 5922 tablet_bootstrap.cc:492] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:25:19.466159 5922 tablet_bootstrap.cc:492] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: Bootstrap complete.
I20260806 09:25:19.468416 5922 ts_tablet_manager.cc:1397] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: Time spent bootstrapping tablet: real 0.053s user 0.033s sys 0.016s
I20260806 09:25:19.471264 5844 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:39089
I20260806 09:25:19.471318 6076 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:39089 every 8 connection(s)
I20260806 09:25:19.473222 5844 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:25:19.475308 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 5844
I20260806 09:25:19.476517 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:34417
--local_ip_for_outbound_sockets=127.16.86.130
--tserver_master_addrs=127.16.86.190:33769
--webserver_port=44299
--webserver_interface=127.16.86.130
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:25:19.478370 5922 raft_consensus.cc:348] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:19.480311 5922 raft_consensus.cc:729] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26e3a0be0611487c8dc56ee0e0863f6d, State: Initialized, Role: FOLLOWER
I20260806 09:25:19.481436 5922 consensus_queue.cc:260] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:19.484198 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:19.477155 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:40805)
0806 09:25:19.482792 (+ 5637us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:19.483171 (+ 379us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":5279,"thread_start_us":4981,"threads_started":1}
W20260806 09:25:19.484988 6078 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:33769 (0 consecutive failures): Network error: Failed to ping master at 127.16.86.190:33769: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
I20260806 09:25:19.486513 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:19.485889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:56869)
0806 09:25:19.486195 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:19.486308 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260806 09:25:19.488740 6080 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:19.487416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:58599)
0806 09:25:19.488324 (+ 908us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:19.488463 (+ 139us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":769,"threads_started":1}
W20260806 09:25:19.491936 6078 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 09:25:19.493311 5922 ts_tablet_manager.cc:1428] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d: Time spent starting tablet: real 0.025s user 0.009s sys 0.013s
I20260806 09:25:19.493744 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:19.492795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:43635)
0806 09:25:19.493101 (+ 306us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:19.493464 (+ 363us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20260806 09:25:19.495242 6082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:19.494594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:35315)
0806 09:25:19.494922 (+ 328us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:19.495052 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":184}
W20260806 09:25:19.880044 6083 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:19.880412 6083 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:19.880607 6083 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:19.888376 6083 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:19.888550 6083 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:19.888747 6083 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:25:19.899025 6083 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:34417
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=44299
--tserver_master_addrs=127.16.86.190:33769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:19.900561 6083 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:19.902069 6083 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:19.918143 6096 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:19.918249 6091 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:19.918381 6090 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:19.919334 6083 server_base.cc:1034] running on GCE node
I20260806 09:25:19.920316 6083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:19.921775 6083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:19.923185 6083 hybrid_clock.cc:648] HybridClock initialized: now 1786008319923108 us; error 42 us; skew 500 ppm
I20260806 09:25:19.923696 6083 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:19 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008319, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:19.928968 6083 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:25:19.931209 6083 webserver.cc:460] Webserver started at http://127.16.86.130:44299/ using document root <none> and password file <none>
I20260806 09:25:19.932195 6083 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:19.932380 6083 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:19.940523 6083 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 09:25:19.945608 6105 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:19.948203 6083 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260806 09:25:19.948529 6083 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "42e16385638a4d5b9655f89d5877a4c7"
format_stamp: "Formatted at 2026-08-06 09:25:12 on dist-test-slave-h3ft"
server_key: "cf283098b82044fbb6a46ce61fa8ecca"
server_key_iv: "684ef4a1e5f7e50fdcd91b22f0e49dc6"
server_key_version: "encryptionkey@0"
I20260806 09:25:19.950006 6083 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:19.978222 6083 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:19.984450 6083 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:19.984848 6083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:19.986368 6083 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:25:19.988418 6083 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:25:19.988540 6083 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:19.988744 6083 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:25:19.988863 6083 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:20.037652 6083 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:34417
I20260806 09:25:20.037746 6254 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:34417 every 8 connection(s)
I20260806 09:25:20.040174 6083 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:25:20.042833 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 6083
I20260806 09:25:20.043998 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:38173
--local_ip_for_outbound_sockets=127.16.86.131
--tserver_master_addrs=127.16.86.190:33769
--webserver_port=40153
--webserver_interface=127.16.86.131
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:42349
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:25:20.047873 6259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:20.044908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:51031)
0806 09:25:20.046132 (+ 1224us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:20.046515 (+ 383us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":820,"thread_start_us":486,"threads_started":1}
W20260806 09:25:20.048805 6255 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:33769 (0 consecutive failures): Network error: Failed to ping master at 127.16.86.190:33769: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
I20260806 09:25:20.050107 6259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:20.049668 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:44079)
0806 09:25:20.049882 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:20.049983 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20260806 09:25:20.051301 6259 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:20.050884 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:39005)
0806 09:25:20.051072 (+ 188us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:20.051178 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 09:25:20.051592 6255 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 09:25:20.498060 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:20.496947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:56999)
0806 09:25:20.497684 (+ 737us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:20.497807 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":592,"thread_start_us":381,"threads_started":1}
W20260806 09:25:20.508165 6260 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:20.508635 6260 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:20.508823 6260 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:20.517741 6260 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:20.517927 6260 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:20.518136 6260 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:25:20.531206 6260 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:42349
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:38173
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=40153
--tserver_master_addrs=127.16.86.190:33769
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:20.533234 6260 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:20.535215 6260 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:20.554296 6273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:20.554365 6269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:20.554242 6267 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:20.557634 6260 server_base.cc:1034] running on GCE node
I20260806 09:25:20.558745 6260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:20.560387 6260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:20.561705 6260 hybrid_clock.cc:648] HybridClock initialized: now 1786008320561636 us; error 40 us; skew 500 ppm
I20260806 09:25:20.562196 6260 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:20 dist-test-slave-h3ft krb5kdc[4262](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008320, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:20.567376 6260 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:25:20.569842 6260 webserver.cc:460] Webserver started at http://127.16.86.131:40153/ using document root <none> and password file <none>
I20260806 09:25:20.570847 6260 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:20.571039 6260 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:20.581358 6260 fs_manager.cc:709] Time spent opening directory manager: real 0.007s user 0.007s sys 0.001s
I20260806 09:25:20.588039 6280 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:20.590128 6260 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260806 09:25:20.590406 6260 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "633faae4693e45bcbe46babba1396b2a"
format_stamp: "Formatted at 2026-08-06 09:25:13 on dist-test-slave-h3ft"
server_key: "a140f6feb89dfebe83412ba1df2547c3"
server_key_iv: "eca9f49c0ad008dfe4c2ddbaadf25095"
server_key_version: "encryptionkey@0"
I20260806 09:25:20.591840 6260 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:20.611385 6260 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:20.617043 6260 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:20.617406 6260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:20.618757 6260 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:25:20.620587 6260 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:25:20.620693 6260 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:20.620831 6260 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:25:20.620914 6260 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:20.664518 6260 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:38173
I20260806 09:25:20.664593 6432 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:38173 every 8 connection(s)
I20260806 09:25:20.666538 6260 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:25:20.672622 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 6260
I20260806 09:25:20.673800 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:20.670508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:45625)
0806 09:25:20.671574 (+ 1066us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:20.671852 (+ 278us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":731,"thread_start_us":428,"threads_started":1}
W20260806 09:25:20.674921 6433 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:33769 (0 consecutive failures): Network error: Failed to ping master at 127.16.86.190:33769: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
I20260806 09:25:20.676255 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:20.675747 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:33391)
0806 09:25:20.675972 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:20.676089 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
I20260806 09:25:20.677405 6435 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:20.677032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:53785)
0806 09:25:20.677184 (+ 152us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:20.677268 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 09:25:20.677862 6433 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 09:25:20.846876 6438 raft_consensus.cc:482] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:25:20.847394 6438 raft_consensus.cc:504] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:20.850142 6438 leader_election.cc:288] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417), 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173)
Aug 06 09:25:20 dist-test-slave-h3ft krb5kdc[4262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008319, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
Aug 06 09:25:20 dist-test-slave-h3ft krb5kdc[4262](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008319, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:25:20.875944 6442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:20.852389 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:42455 (local address 127.16.86.131:38173)
0806 09:25:20.853173 (+ 784us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:20.853184 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:20.853224 (+ 40us) server_negotiation.cc:407] Connection header received
0806 09:25:20.854942 (+ 1718us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:20.854953 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:20.855497 (+ 544us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:20.855937 (+ 440us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:20.858066 (+ 2129us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:20.859478 (+ 1412us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:20.861414 (+ 1936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:20.862082 (+ 668us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:20.862292 (+ 210us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:20.866911 (+ 4619us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:20.866925 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:20.866952 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:20.867059 (+ 107us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:20.870825 (+ 3766us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:20.871880 (+ 1055us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:20.871892 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:20.871904 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:20.872130 (+ 226us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:20.872662 (+ 532us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:20.872670 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:20.872702 (+ 32us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:20.873549 (+ 847us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:20.873760 (+ 211us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:20.874852 (+ 1092us) server_negotiation.cc:299] Negotiation successful
0806 09:25:20.875418 (+ 566us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":617,"thread_start_us":412,"threads_started":1}
I20260806 09:25:20.876050 6441 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:20.851172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:50761 (local address 127.16.86.130:34417)
0806 09:25:20.852574 (+ 1402us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:20.852590 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:20.852631 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:25:20.854352 (+ 1721us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:20.854362 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:20.854948 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:20.855468 (+ 520us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:20.858119 (+ 2651us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:20.859514 (+ 1395us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:20.861437 (+ 1923us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:20.862206 (+ 769us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:20.862412 (+ 206us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:20.867443 (+ 5031us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:20.867454 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:20.867470 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:20.867526 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:20.871092 (+ 3566us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:20.871979 (+ 887us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:20.871986 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:20.871992 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:20.872178 (+ 186us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:20.872739 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:20.872772 (+ 33us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:20.872775 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:20.873403 (+ 628us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:20.873557 (+ 154us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:20.874928 (+ 1371us) server_negotiation.cc:299] Negotiation successful
0806 09:25:20.875520 (+ 592us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1201,"thread_start_us":1065,"threads_started":1}
I20260806 09:25:20.876358 6261 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:20.850600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:38173 (local address 127.16.86.129:42455)
0806 09:25:20.850828 (+ 228us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:20.850883 (+ 55us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:20.851576 (+ 693us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:20.855906 (+ 4330us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:20.855911 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:20.856957 (+ 1046us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:20.857856 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:20.857879 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:20.859691 (+ 1812us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:20.859697 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:20.861263 (+ 1566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:20.861277 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:20.862289 (+ 1012us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:20.862296 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:20.862571 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:20.863520 (+ 949us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:20.863558 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:20.866462 (+ 2904us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:20.871055 (+ 4593us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:20.871065 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:20.871088 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:20.871710 (+ 622us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:20.872282 (+ 572us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:20.872286 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:20.872290 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:20.872506 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:20.873708 (+ 1202us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:20.873712 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:20.874153 (+ 441us) client_negotiation.cc:769] Sending connection context
0806 09:25:20.874513 (+ 360us) client_negotiation.cc:240] Negotiation successful
0806 09:25:20.875725 (+ 1212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":79,"mutex_wait_us":259}
I20260806 09:25:20.876761 6440 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:20.850684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:34417 (local address 127.16.86.129:50761)
0806 09:25:20.851639 (+ 955us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:20.851664 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:20.851823 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:20.855618 (+ 3795us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:20.855664 (+ 46us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:20.856959 (+ 1295us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:20.857856 (+ 897us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:20.857880 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:20.859715 (+ 1835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:20.859720 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:20.861259 (+ 1539us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:20.861272 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:20.862664 (+ 1392us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:20.862670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:20.862892 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:20.863768 (+ 876us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:20.863799 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:20.867024 (+ 3225us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:20.871262 (+ 4238us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:20.871266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:20.871269 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:20.871827 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:20.872353 (+ 526us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:20.872357 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:20.872359 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:20.872591 (+ 232us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:20.873582 (+ 991us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:20.873594 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:20.874151 (+ 557us) client_negotiation.cc:769] Sending connection context
0806 09:25:20.875067 (+ 916us) client_negotiation.cc:240] Negotiation successful
0806 09:25:20.875525 (+ 458us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"mutex_wait_us":191,"spinlock_wait_cycles":33408,"thread_start_us":711,"threads_started":1}
W20260806 09:25:20.880296 5916 leader_election.cc:341] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417): Not found: Tablet not found: ee2eadeeacc74c5aa3cffc2c26f85cd4
W20260806 09:25:20.880566 5915 leader_election.cc:341] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173): Not found: Tablet not found: ee2eadeeacc74c5aa3cffc2c26f85cd4
I20260806 09:25:20.880765 5915 leader_election.cc:302] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e3a0be0611487c8dc56ee0e0863f6d; no voters: 42e16385638a4d5b9655f89d5877a4c7, 633faae4693e45bcbe46babba1396b2a
I20260806 09:25:20.881246 6438 raft_consensus.cc:2726] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 09:25:21.053529 6443 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:21.052221 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:53177)
0806 09:25:21.052957 (+ 736us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:21.053175 (+ 218us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":353,"threads_started":1}
I20260806 09:25:21.500275 6446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:21.499420 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:52739)
0806 09:25:21.499980 (+ 560us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:21.500085 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":297,"threads_started":1}
I20260806 09:25:21.679890 6447 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:21.678749 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:58205)
0806 09:25:21.679461 (+ 712us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:21.679658 (+ 197us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":374,"threads_started":1}
I20260806 09:25:22.055876 6448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:22.054811 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:41209)
0806 09:25:22.055567 (+ 756us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:22.055682 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":405,"threads_started":1}
I20260806 09:25:22.502508 6450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:22.501560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:58645)
0806 09:25:22.502200 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:22.502312 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":299,"threads_started":1}
I20260806 09:25:22.568446 6452 raft_consensus.cc:482] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:25:22.568784 6452 raft_consensus.cc:504] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:22.569761 6452 leader_election.cc:288] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417), 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173)
W20260806 09:25:22.571115 5916 leader_election.cc:341] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417): Not found: Tablet not found: ee2eadeeacc74c5aa3cffc2c26f85cd4
W20260806 09:25:22.571419 5915 leader_election.cc:341] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173): Not found: Tablet not found: ee2eadeeacc74c5aa3cffc2c26f85cd4
I20260806 09:25:22.571521 5915 leader_election.cc:302] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e3a0be0611487c8dc56ee0e0863f6d; no voters: 42e16385638a4d5b9655f89d5877a4c7, 633faae4693e45bcbe46babba1396b2a
I20260806 09:25:22.571913 6452 raft_consensus.cc:2726] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 09:25:22.682299 6454 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:22.681216 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:35169)
0806 09:25:22.681882 (+ 666us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:22.682015 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":521,"thread_start_us":330,"threads_started":1}
I20260806 09:25:23.058068 6455 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:23.057075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:52919)
0806 09:25:23.057742 (+ 667us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:23.057863 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":327,"threads_started":1}
I20260806 09:25:23.504890 6456 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:23.503888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:59751)
0806 09:25:23.504509 (+ 621us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:23.504642 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":331,"threads_started":1}
I20260806 09:25:23.684849 6457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:23.683798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:40025)
0806 09:25:23.684462 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:23.684635 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":523,"thread_start_us":361,"threads_started":1}
I20260806 09:25:24.060739 6458 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:24.059505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:56337)
0806 09:25:24.060274 (+ 769us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:24.060419 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":402,"threads_started":1}
I20260806 09:25:24.413017 6459 raft_consensus.cc:482] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:25:24.413316 6459 raft_consensus.cc:504] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:24.414206 6459 leader_election.cc:288] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417), 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173)
W20260806 09:25:24.415614 5915 leader_election.cc:341] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173): Not found: Tablet not found: ee2eadeeacc74c5aa3cffc2c26f85cd4
W20260806 09:25:24.416010 5916 leader_election.cc:341] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417): Not found: Tablet not found: ee2eadeeacc74c5aa3cffc2c26f85cd4
I20260806 09:25:24.416234 5916 leader_election.cc:302] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e3a0be0611487c8dc56ee0e0863f6d; no voters: 42e16385638a4d5b9655f89d5877a4c7, 633faae4693e45bcbe46babba1396b2a
I20260806 09:25:24.416623 6459 raft_consensus.cc:2726] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 09:25:24.507510 6460 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:24.506393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:51763)
0806 09:25:24.507106 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:24.507250 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":294,"threads_started":1}
I20260806 09:25:24.687184 6461 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:24.686172 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:50607)
0806 09:25:24.686831 (+ 659us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:24.686933 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":374,"threads_started":1}
I20260806 09:25:25.063129 6462 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:25.062045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:56241)
0806 09:25:25.062739 (+ 694us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:25.062849 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":562,"thread_start_us":357,"threads_started":1}
I20260806 09:25:25.509536 6463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:25.508669 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:33991)
0806 09:25:25.509255 (+ 586us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:25.509347 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":314,"threads_started":1}
I20260806 09:25:25.689538 6464 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:25.688532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:60927)
0806 09:25:25.689186 (+ 654us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:25.689331 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":344,"threads_started":1}
I20260806 09:25:26.065642 6465 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:26.064409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:59275)
0806 09:25:26.065250 (+ 841us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:26.065392 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":515,"threads_started":1}
I20260806 09:25:26.512051 6467 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:26.511015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:36853)
0806 09:25:26.511730 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:26.511830 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":573,"thread_start_us":415,"threads_started":1}
I20260806 09:25:26.692209 6469 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:26.691190 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:39913)
0806 09:25:26.691820 (+ 630us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:26.691952 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":493,"thread_start_us":314,"threads_started":1}
I20260806 09:25:27.068653 6470 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:27.067305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:51639)
0806 09:25:27.068277 (+ 972us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:27.068403 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":468,"threads_started":1}
I20260806 09:25:27.514209 6473 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:27.513192 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:43145)
0806 09:25:27.513856 (+ 664us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:27.513973 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":339,"threads_started":1}
I20260806 09:25:27.694941 6474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:27.693534 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:43851)
0806 09:25:27.694464 (+ 930us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:27.694606 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":457,"threads_started":1}
I20260806 09:25:28.071238 6475 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:28.070057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:40671)
0806 09:25:28.070680 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:28.070817 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":316,"threads_started":1}
I20260806 09:25:28.516697 6476 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:28.515723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:51613)
0806 09:25:28.516321 (+ 598us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:28.516448 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":315,"threads_started":1}
I20260806 09:25:28.697482 6479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:28.696272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:38625)
0806 09:25:28.697127 (+ 855us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:28.697236 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":719,"thread_start_us":550,"threads_started":1}
I20260806 09:25:28.991729 6481 raft_consensus.cc:482] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:25:28.992115 6481 raft_consensus.cc:504] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:28.993047 6481 leader_election.cc:288] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417), 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173)
W20260806 09:25:28.994498 5915 leader_election.cc:341] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173): Not found: Tablet not found: ee2eadeeacc74c5aa3cffc2c26f85cd4
W20260806 09:25:28.994856 5916 leader_election.cc:341] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417): Not found: Tablet not found: ee2eadeeacc74c5aa3cffc2c26f85cd4
I20260806 09:25:28.995079 5916 leader_election.cc:302] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e3a0be0611487c8dc56ee0e0863f6d; no voters: 42e16385638a4d5b9655f89d5877a4c7, 633faae4693e45bcbe46babba1396b2a
I20260806 09:25:28.995435 6481 raft_consensus.cc:2726] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 09:25:29.074172 6483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:29.072896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:60513)
0806 09:25:29.073658 (+ 762us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:29.073822 (+ 164us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":348,"threads_started":1}
I20260806 09:25:29.519260 6484 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:29.518193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:49527)
0806 09:25:29.518863 (+ 670us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:29.518979 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":333,"threads_started":1}
I20260806 09:25:29.699735 6486 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:29.698603 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:45449)
0806 09:25:29.699387 (+ 784us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:29.699518 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":620,"thread_start_us":404,"threads_started":1}
I20260806 09:25:30.076692 6487 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:30.075653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:55689)
0806 09:25:30.076308 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:30.076444 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":379,"threads_started":1}
I20260806 09:25:30.521427 6488 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:30.520521 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:48955)
0806 09:25:30.521126 (+ 605us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:30.521234 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":478,"thread_start_us":318,"threads_started":1}
I20260806 09:25:30.702260 6491 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:30.701185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:38253)
0806 09:25:30.701872 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:30.702004 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":362,"threads_started":1}
I20260806 09:25:31.079239 6492 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:31.078148 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:45975)
0806 09:25:31.078816 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:31.078948 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":315,"threads_started":1}
I20260806 09:25:31.523770 6494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:31.522713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:55317)
0806 09:25:31.523456 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:31.523572 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":450,"threads_started":1}
I20260806 09:25:31.704452 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:31.703455 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:54815)
0806 09:25:31.704115 (+ 660us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:31.704257 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":335,"threads_started":1}
I20260806 09:25:32.081561 6496 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:32.080554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:52253)
0806 09:25:32.081205 (+ 651us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:32.081328 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":348,"threads_started":1}
I20260806 09:25:32.526312 6497 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:32.525060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:53377)
0806 09:25:32.525914 (+ 854us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:32.526072 (+ 158us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":631,"thread_start_us":397,"threads_started":1}
I20260806 09:25:32.706976 6499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:32.705839 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:55557)
0806 09:25:32.706610 (+ 771us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:32.706746 (+ 136us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":445,"threads_started":1}
I20260806 09:25:33.074389 6500 raft_consensus.cc:482] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:25:33.074715 6500 raft_consensus.cc:504] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "42e16385638a4d5b9655f89d5877a4c7" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 34417 } } peers { permanent_uuid: "633faae4693e45bcbe46babba1396b2a" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38173 } } peers { permanent_uuid: "26e3a0be0611487c8dc56ee0e0863f6d" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 39089 } }
I20260806 09:25:33.075641 6500 leader_election.cc:288] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417), 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173)
W20260806 09:25:33.077073 5916 leader_election.cc:341] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 42e16385638a4d5b9655f89d5877a4c7 (127.16.86.130:34417): Not found: Tablet not found: ee2eadeeacc74c5aa3cffc2c26f85cd4
W20260806 09:25:33.077314 5915 leader_election.cc:341] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 633faae4693e45bcbe46babba1396b2a (127.16.86.131:38173): Not found: Tablet not found: ee2eadeeacc74c5aa3cffc2c26f85cd4
I20260806 09:25:33.077514 5915 leader_election.cc:302] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 26e3a0be0611487c8dc56ee0e0863f6d; no voters: 42e16385638a4d5b9655f89d5877a4c7, 633faae4693e45bcbe46babba1396b2a
I20260806 09:25:33.077936 6500 raft_consensus.cc:2726] T ee2eadeeacc74c5aa3cffc2c26f85cd4 P 26e3a0be0611487c8dc56ee0e0863f6d [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260806 09:25:33.083758 6501 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:33.082762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:35997)
0806 09:25:33.083463 (+ 701us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:33.083577 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":441,"threads_started":1}
I20260806 09:25:33.529086 6502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:33.527893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:48753)
0806 09:25:33.528648 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:33.528793 (+ 145us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":338,"threads_started":1}
I20260806 09:25:33.709394 6503 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:33.708434 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:39319)
0806 09:25:33.709081 (+ 647us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:33.709207 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":520,"thread_start_us":357,"threads_started":1}
I20260806 09:25:34.086123 6505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:34.085043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:58773)
0806 09:25:34.085745 (+ 702us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:34.085846 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":362,"threads_started":1}
I20260806 09:25:34.531813 6507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:34.530678 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:56459)
0806 09:25:34.531421 (+ 743us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:34.531553 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":421,"threads_started":1}
I20260806 09:25:34.711874 6509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:34.710780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:34587)
0806 09:25:34.711524 (+ 744us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:34.711642 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":616,"thread_start_us":436,"threads_started":1}
I20260806 09:25:35.088918 6510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:35.087913 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.130:52701)
0806 09:25:35.088553 (+ 640us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:35.088695 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":501,"thread_start_us":324,"threads_started":1}
I20260806 09:25:35.534128 6511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:35.532971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.129:54115)
0806 09:25:35.533726 (+ 755us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:35.533902 (+ 176us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":599,"thread_start_us":371,"threads_started":1}
I20260806 09:25:35.679432 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 5844
I20260806 09:25:35.699513 6513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:35.681183 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37188 (local address 127.16.86.129:39089)
0806 09:25:35.682113 (+ 930us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:35.682123 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:35.682188 (+ 65us) server_negotiation.cc:407] Connection header received
0806 09:25:35.682299 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:35.682314 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:35.682728 (+ 414us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:35.682984 (+ 256us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:35.684722 (+ 1738us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:35.686161 (+ 1439us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:35.687640 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:35.688415 (+ 775us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:35.688549 (+ 134us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:35.692014 (+ 3465us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:35.692031 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:35.692054 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:35.692108 (+ 54us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:35.695936 (+ 3828us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:35.696941 (+ 1005us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:35.696951 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:35.696954 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:35.697221 (+ 267us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:35.697609 (+ 388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:35.697613 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:35.697615 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:35.698127 (+ 512us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:35.698412 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:35.698711 (+ 299us) server_negotiation.cc:299] Negotiation successful
0806 09:25:35.699140 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":660,"thread_start_us":454,"threads_started":1}
I20260806 09:25:35.701061 6069 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:35.714522 6516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:35.713554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:33769 (local address 127.16.86.131:49161)
0806 09:25:35.714179 (+ 625us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:35.714345 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:33769: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":313,"threads_started":1}
I20260806 09:25:35.805042 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 5844
I20260806 09:25:35.837389 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 6083
I20260806 09:25:35.853650 6517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:35.838450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49698 (local address 127.16.86.130:34417)
0806 09:25:35.839195 (+ 745us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:35.839200 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:35.839224 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:25:35.839303 (+ 79us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:35.839307 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:35.839384 (+ 77us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:35.839508 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:35.841081 (+ 1573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:35.842061 (+ 980us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:35.843554 (+ 1493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:35.844059 (+ 505us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:35.844181 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:35.846983 (+ 2802us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:35.846988 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:35.846992 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:35.847102 (+ 110us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:35.850355 (+ 3253us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:35.851316 (+ 961us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:35.851323 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:35.851325 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:35.851519 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:35.851928 (+ 409us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:35.851933 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:35.851939 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:35.852279 (+ 340us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:35.852631 (+ 352us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:35.852918 (+ 287us) server_negotiation.cc:299] Negotiation successful
0806 09:25:35.853243 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":620,"thread_start_us":361,"threads_started":1}
I20260806 09:25:35.854597 6249 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:35.947106 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 6083
I20260806 09:25:35.976292 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 6260
I20260806 09:25:35.993613 6519 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:35.977297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55444 (local address 127.16.86.131:38173)
0806 09:25:35.978113 (+ 816us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:35.978119 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:35.978138 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:25:35.978212 (+ 74us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:35.978216 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:35.978297 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:35.978446 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:35.980235 (+ 1789us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:35.981269 (+ 1034us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:35.982651 (+ 1382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:35.983261 (+ 610us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:35.983421 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:35.986883 (+ 3462us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:35.986890 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:35.986894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:35.986936 (+ 42us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:35.990358 (+ 3422us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:35.991265 (+ 907us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:35.991269 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:35.991272 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:35.991431 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:35.991866 (+ 435us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:35.991873 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:35.991881 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:35.992208 (+ 327us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:35.992597 (+ 389us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:35.992885 (+ 288us) server_negotiation.cc:299] Negotiation successful
0806 09:25:35.993156 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":657,"thread_start_us":380,"threads_started":1}
I20260806 09:25:35.994490 6427 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:36.086208 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 6260
2026-08-06T09:25:36Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (26874 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:25:36 dist-test-slave-h3ft krb5kdc[6525](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:25:36 dist-test-slave-h3ft krb5kdc[6525](info): set up 2 sockets
Aug 06 09:25:36 dist-test-slave-h3ft krb5kdc[6525](info): commencing operation
krb5kdc: starting...
W20260806 09:25:37.467808 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.323s user 0.004s sys 0.045s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:25:37 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:25:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:25:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:25:37.840750 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:34327
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:34327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:25:38.252918 6539 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:38.253314 6539 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:38.253479 6539 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:38.262099 6539 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 09:25:38.262239 6539 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 09:25:38.262287 6539 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 09:25:38.262411 6539 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:25:38.262526 6539 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:38.262662 6539 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:38.262737 6539 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:25:38.262852 6539 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:25:38.274349 6539 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:34327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:38.275998 6539 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:38.277451 6539 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:38.292095 6547 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:38.292610 6548 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:38.292212 6550 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:38.293455 6539 server_base.cc:1034] running on GCE node
I20260806 09:25:38.294412 6539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:38.296085 6539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:38.297356 6539 hybrid_clock.cc:648] HybridClock initialized: now 1786008338297279 us; error 41 us; skew 500 ppm
I20260806 09:25:38.297832 6539 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:38 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008338, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:38.301980 6539 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:25:38.304105 6539 webserver.cc:460] Webserver started at http://127.16.86.190:39097/ using document root <none> and password file <none>
I20260806 09:25:38.304975 6539 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:38.305130 6539 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:38.305529 6539 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:25:38.309144 6539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "e063cec8fc994c39af3bc0a6b3d75ffa"
format_stamp: "Formatted at 2026-08-06 09:25:38 on dist-test-slave-h3ft"
server_key: "a2cfafb68656cf53cdc4ac7d524d1c9f"
server_key_iv: "c464e578db4287f2116d86c96412b70a"
server_key_version: "encryptionkey@0"
I20260806 09:25:38.310140 6539 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "e063cec8fc994c39af3bc0a6b3d75ffa"
format_stamp: "Formatted at 2026-08-06 09:25:38 on dist-test-slave-h3ft"
server_key: "a2cfafb68656cf53cdc4ac7d524d1c9f"
server_key_iv: "c464e578db4287f2116d86c96412b70a"
server_key_version: "encryptionkey@0"
I20260806 09:25:38.316676 6539 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:25:38.321089 6557 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:38.322804 6539 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:25:38.323109 6539 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "e063cec8fc994c39af3bc0a6b3d75ffa"
format_stamp: "Formatted at 2026-08-06 09:25:38 on dist-test-slave-h3ft"
server_key: "a2cfafb68656cf53cdc4ac7d524d1c9f"
server_key_iv: "c464e578db4287f2116d86c96412b70a"
server_key_version: "encryptionkey@0"
I20260806 09:25:38.323393 6539 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:38.348438 6539 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:38.353443 6539 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:38.353767 6539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:38.376143 6539 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:34327
I20260806 09:25:38.376187 6623 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:34327 every 8 connection(s)
I20260806 09:25:38.377722 6539 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:25:38.381978 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 6539
I20260806 09:25:38.382259 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:25:38.383277 16730 external_mini_cluster.cc:1428] Setting key 88e5859cac7ce579e7ee8657786736b5
I20260806 09:25:38.383394 6624 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:38.401715 6624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Bootstrap starting.
Aug 06 09:25:38 dist-test-slave-h3ft krb5kdc[6525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:38.406424 6624 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:38.407930 6624 log.cc:824] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:38.411852 6624 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: No bootstrap required, opened a new log
I20260806 09:25:38.412199 6628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:38.386022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55582 (local address 127.16.86.190:34327)
0806 09:25:38.386973 (+ 951us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:38.386986 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:38.387101 (+ 115us) server_negotiation.cc:407] Connection header received
0806 09:25:38.389398 (+ 2297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:38.389422 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:38.389863 (+ 441us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:38.390238 (+ 375us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:38.392517 (+ 2279us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:38.394028 (+ 1511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:38.396220 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:38.397015 (+ 795us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:38.397217 (+ 202us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:38.403368 (+ 6151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:38.403380 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:38.403401 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:38.403466 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:38.407243 (+ 3777us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:38.408166 (+ 923us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:38.408174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:38.408181 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:38.408400 (+ 219us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:38.408872 (+ 472us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:38.408876 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:38.408879 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:38.409540 (+ 661us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:38.409897 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:38.410420 (+ 523us) server_negotiation.cc:299] Negotiation successful
0806 09:25:38.410911 (+ 491us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":449,"threads_started":1}
I20260806 09:25:38.421583 6624 raft_consensus.cc:348] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } }
I20260806 09:25:38.422112 6624 raft_consensus.cc:374] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:38.422236 6624 raft_consensus.cc:729] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063cec8fc994c39af3bc0a6b3d75ffa, State: Initialized, Role: FOLLOWER
I20260806 09:25:38.423130 6624 consensus_queue.cc:260] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } }
I20260806 09:25:38.423559 6624 raft_consensus.cc:388] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:25:38.423768 6624 raft_consensus.cc:482] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:25:38.424011 6624 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:38.426657 6624 raft_consensus.cc:504] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } }
I20260806 09:25:38.427409 6624 leader_election.cc:302] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e063cec8fc994c39af3bc0a6b3d75ffa; no voters:
I20260806 09:25:38.428282 6624 leader_election.cc:288] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:25:38.428490 6630 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:25:38.429436 6630 raft_consensus.cc:686] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 1 LEADER]: Becoming Leader. State: Replica: e063cec8fc994c39af3bc0a6b3d75ffa, State: Running, Role: LEADER
I20260806 09:25:38.430138 6630 consensus_queue.cc:237] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } }
I20260806 09:25:38.431121 6624 sys_catalog.cc:564] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:25:38.433566 6631 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } } }
I20260806 09:25:38.433565 6633 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e063cec8fc994c39af3bc0a6b3d75ffa. Latest consensus state: current_term: 1 leader_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } } }
I20260806 09:25:38.434206 6631 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:38.434203 6633 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: This master's current role is: LEADER
I20260806 09:25:38.435382 6638 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:25:38.442759 6638 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:25:38.453685 6638 catalog_manager.cc:1347] Generated new cluster ID: 6e87a7edfa5a4e62865e50dd6680ff96
I20260806 09:25:38.453878 6638 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:25:38.479218 6638 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:25:38.480870 6638 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:25:38.494822 6638 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Generated new TSK 0
I20260806 09:25:38.495885 6638 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:25:38.584237 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:34327
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:25:38.956239 6657 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:38.956573 6657 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:38.956754 6657 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:38.964730 6657 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:38.964870 6657 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:38.965062 6657 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:25:38.975658 6657 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:38.977041 6657 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:38.978384 6657 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:38.994082 6662 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:38.994684 6665 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:38.994773 6663 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:38.995410 6657 server_base.cc:1034] running on GCE node
I20260806 09:25:38.996179 6657 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:38.997318 6657 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:38.998621 6657 hybrid_clock.cc:648] HybridClock initialized: now 1786008338998553 us; error 54 us; skew 500 ppm
I20260806 09:25:38.999085 6657 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:39 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008339, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:39.007872 6657 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:25:39.009723 6657 webserver.cc:460] Webserver started at http://127.16.86.129:39767/ using document root <none> and password file <none>
I20260806 09:25:39.010443 6657 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:39.010555 6657 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:39.010836 6657 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:25:39.014040 6657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "4d6da66d904e45219f37ff127f9dc2e4"
format_stamp: "Formatted at 2026-08-06 09:25:39 on dist-test-slave-h3ft"
server_key: "3bb55387eae8f12ce29f3bfb0897e06b"
server_key_iv: "02b7e1de7c82950ee2880d78645e3666"
server_key_version: "encryptionkey@0"
I20260806 09:25:39.014791 6657 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "4d6da66d904e45219f37ff127f9dc2e4"
format_stamp: "Formatted at 2026-08-06 09:25:39 on dist-test-slave-h3ft"
server_key: "3bb55387eae8f12ce29f3bfb0897e06b"
server_key_iv: "02b7e1de7c82950ee2880d78645e3666"
server_key_version: "encryptionkey@0"
I20260806 09:25:39.020771 6657 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.001s
I20260806 09:25:39.024999 6673 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:39.026777 6657 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:25:39.027041 6657 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "4d6da66d904e45219f37ff127f9dc2e4"
format_stamp: "Formatted at 2026-08-06 09:25:39 on dist-test-slave-h3ft"
server_key: "3bb55387eae8f12ce29f3bfb0897e06b"
server_key_iv: "02b7e1de7c82950ee2880d78645e3666"
server_key_version: "encryptionkey@0"
I20260806 09:25:39.027297 6657 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:39.057535 6657 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:39.063236 6657 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:39.063664 6657 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:39.064998 6657 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:25:39.066795 6657 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:25:39.066881 6657 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:39.067137 6657 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:25:39.067222 6657 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:39.108047 6657 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:43931
I20260806 09:25:39.108117 6822 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:43931 every 8 connection(s)
I20260806 09:25:39.110270 6657 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:25:39.119253 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 6657
I20260806 09:25:39.119545 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:25:39.120514 16730 external_mini_cluster.cc:1428] Setting key 119f79adc0c2db06c8b511d122bdca41
Aug 06 09:25:39 dist-test-slave-h3ft krb5kdc[6525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008339, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:39.137887 6826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:39.113810 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:48703 (local address 127.16.86.190:34327)
0806 09:25:39.114435 (+ 625us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:39.114443 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:39.115741 (+ 1298us) server_negotiation.cc:407] Connection header received
0806 09:25:39.117783 (+ 2042us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:39.117789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:39.117927 (+ 138us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:39.118078 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:39.120134 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:39.121332 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:39.123524 (+ 2192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:39.124247 (+ 723us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:39.124432 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:39.128811 (+ 4379us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:39.128821 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:39.128830 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:39.128885 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:39.133459 (+ 4574us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:39.134481 (+ 1022us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:39.134490 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:39.134497 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:39.134696 (+ 199us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:39.135264 (+ 568us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:39.135272 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:39.135278 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:39.135775 (+ 497us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:39.135961 (+ 186us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:39.136979 (+ 1018us) server_negotiation.cc:299] Negotiation successful
0806 09:25:39.137315 (+ 336us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":446,"thread_start_us":294,"threads_started":1}
I20260806 09:25:39.138718 6828 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:39.114524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:48703)
0806 09:25:39.115529 (+ 1005us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:39.115587 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:39.117460 (+ 1873us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:39.118376 (+ 916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:39.118385 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:39.119138 (+ 753us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:39.119915 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:39.119935 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:39.121566 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:39.121571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:39.123349 (+ 1778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:39.123373 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:39.124540 (+ 1167us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:39.124545 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:39.124752 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:39.125555 (+ 803us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:39.125580 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:39.128480 (+ 2900us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:39.133692 (+ 5212us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:39.133701 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:39.133715 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:39.134286 (+ 571us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:39.134828 (+ 542us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:39.134831 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:39.134833 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:39.135093 (+ 260us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:39.135960 (+ 867us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:39.135967 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:39.136396 (+ 429us) client_negotiation.cc:769] Sending connection context
0806 09:25:39.136768 (+ 372us) client_negotiation.cc:240] Negotiation successful
0806 09:25:39.137315 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":710,"thread_start_us":425,"threads_started":1}
I20260806 09:25:39.140899 6823 heartbeater.cc:347] Connected to a master server at 127.16.86.190:34327
I20260806 09:25:39.141278 6823 heartbeater.cc:464] Registering TS with master...
I20260806 09:25:39.142168 6823 heartbeater.cc:511] Master 127.16.86.190:34327 requested a full tablet report, sending...
I20260806 09:25:39.145177 6578 ts_manager.cc:194] Registered new tserver with Master: 4d6da66d904e45219f37ff127f9dc2e4 (127.16.86.129:43931)
I20260806 09:25:39.146845 6578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:48703
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:25:39.211750 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:34327
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:25:39.632339 6832 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:39.632808 6832 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:39.633006 6832 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:39.641840 6832 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:39.642056 6832 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:39.642247 6832 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:25:39.654182 6832 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:39.656598 6832 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:39.658627 6832 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:39.677762 6844 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:39.677847 6840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:39.677821 6841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:39.678399 6832 server_base.cc:1034] running on GCE node
I20260806 09:25:39.679577 6832 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:39.680943 6832 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:39.682233 6832 hybrid_clock.cc:648] HybridClock initialized: now 1786008339682166 us; error 59 us; skew 500 ppm
I20260806 09:25:39.682655 6832 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:39 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008339, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:39.686709 6832 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:25:39.688809 6832 webserver.cc:460] Webserver started at http://127.16.86.130:45413/ using document root <none> and password file <none>
I20260806 09:25:39.689641 6832 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:39.689821 6832 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:39.690171 6832 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:25:39.693547 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "398206deeb46492e87d93151bcf0b252"
format_stamp: "Formatted at 2026-08-06 09:25:39 on dist-test-slave-h3ft"
server_key: "826cb9293b0c4a2ed89f6ac7614c4239"
server_key_iv: "7442e7b4e151955c9327a4e2ff1fda8f"
server_key_version: "encryptionkey@0"
I20260806 09:25:39.694424 6832 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "398206deeb46492e87d93151bcf0b252"
format_stamp: "Formatted at 2026-08-06 09:25:39 on dist-test-slave-h3ft"
server_key: "826cb9293b0c4a2ed89f6ac7614c4239"
server_key_iv: "7442e7b4e151955c9327a4e2ff1fda8f"
server_key_version: "encryptionkey@0"
I20260806 09:25:39.700851 6832 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260806 09:25:39.705672 6856 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:39.707613 6832 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:25:39.707895 6832 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "398206deeb46492e87d93151bcf0b252"
format_stamp: "Formatted at 2026-08-06 09:25:39 on dist-test-slave-h3ft"
server_key: "826cb9293b0c4a2ed89f6ac7614c4239"
server_key_iv: "7442e7b4e151955c9327a4e2ff1fda8f"
server_key_version: "encryptionkey@0"
I20260806 09:25:39.708166 6832 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:39.743088 6832 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:39.748230 6832 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:39.748581 6832 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:39.749810 6832 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:25:39.751544 6832 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:25:39.751658 6832 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:39.751834 6832 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:25:39.751914 6832 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:39.794960 6832 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:41903
I20260806 09:25:39.795208 6994 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:41903 every 8 connection(s)
I20260806 09:25:39.796470 6832 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:25:39.802492 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 6832
I20260806 09:25:39.802757 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:25:39.803436 16730 external_mini_cluster.cc:1428] Setting key a846930311266004f2b540ed4b666813
Aug 06 09:25:39 dist-test-slave-h3ft krb5kdc[6525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008339, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:39.821981 6999 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:39.799978 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:46717 (local address 127.16.86.190:34327)
0806 09:25:39.800488 (+ 510us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:39.800493 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:39.801773 (+ 1280us) server_negotiation.cc:407] Connection header received
0806 09:25:39.804088 (+ 2315us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:39.804094 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:39.804209 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:39.804341 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:39.806327 (+ 1986us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:39.807166 (+ 839us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:39.808757 (+ 1591us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:39.809177 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:39.809314 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:39.814253 (+ 4939us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:39.814258 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:39.814262 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:39.814300 (+ 38us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:39.817420 (+ 3120us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:39.818805 (+ 1385us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:39.818810 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:39.818812 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:39.818958 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:39.819753 (+ 795us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:39.819757 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:39.819760 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:39.820104 (+ 344us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:39.820248 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:39.821476 (+ 1228us) server_negotiation.cc:299] Negotiation successful
0806 09:25:39.821677 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":394,"thread_start_us":281,"threads_started":1}
I20260806 09:25:39.823758 7000 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:39.800637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:46717)
0806 09:25:39.801562 (+ 925us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:39.801612 (+ 50us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:39.803772 (+ 2160us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:39.804618 (+ 846us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:39.804626 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:39.805457 (+ 831us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:39.806116 (+ 659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:39.806135 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:39.807338 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:39.807342 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:39.808595 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:39.808607 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:39.809423 (+ 816us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:39.809430 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:39.809663 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:39.810421 (+ 758us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:39.810451 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:39.813882 (+ 3431us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:39.817656 (+ 3774us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:39.817672 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:39.817701 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:39.818625 (+ 924us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:39.819185 (+ 560us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:39.819193 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:39.819199 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:39.819548 (+ 349us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:39.820282 (+ 734us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:39.820295 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:39.820956 (+ 661us) client_negotiation.cc:769] Sending connection context
0806 09:25:39.821371 (+ 415us) client_negotiation.cc:240] Negotiation successful
0806 09:25:39.822149 (+ 778us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":382,"threads_started":1}
I20260806 09:25:39.825912 6995 heartbeater.cc:347] Connected to a master server at 127.16.86.190:34327
I20260806 09:25:39.826359 6995 heartbeater.cc:464] Registering TS with master...
I20260806 09:25:39.827560 6995 heartbeater.cc:511] Master 127.16.86.190:34327 requested a full tablet report, sending...
I20260806 09:25:39.829749 6578 ts_manager.cc:194] Registered new tserver with Master: 398206deeb46492e87d93151bcf0b252 (127.16.86.130:41903)
I20260806 09:25:39.830829 6578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:46717
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:25:39.887949 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:34327
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:25:40.150295 6823 heartbeater.cc:503] Master 127.16.86.190:34327 was elected leader, sending a full tablet report...
W20260806 09:25:40.279402 7006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:25:40.279716 7006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:25:40.279815 7006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:25:40.287179 7006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:25:40.287302 7006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:25:40.287417 7006 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:25:40.297340 7006 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:25:40.298724 7006 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:25:40.300048 7006 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:25:40.315637 7018 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:40.315616 7015 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:25:40.315574 7016 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:25:40.316669 7006 server_base.cc:1034] running on GCE node
I20260806 09:25:40.317373 7006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:25:40.318444 7006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:25:40.319767 7006 hybrid_clock.cc:648] HybridClock initialized: now 1786008340319696 us; error 58 us; skew 500 ppm
I20260806 09:25:40.320152 7006 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:25:40 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008340, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:25:40.324182 7006 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:25:40.326002 7006 webserver.cc:460] Webserver started at http://127.16.86.131:38961/ using document root <none> and password file <none>
I20260806 09:25:40.326767 7006 fs_manager.cc:359] Metadata directory not provided
I20260806 09:25:40.326922 7006 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:25:40.327270 7006 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:25:40.330353 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "40e0f22a71424e07bed66412e6d24f19"
format_stamp: "Formatted at 2026-08-06 09:25:40 on dist-test-slave-h3ft"
server_key: "4306d4d44746e5f12dafdfa827c0536f"
server_key_iv: "88307b3f26b4221c0a35802d1a42c23c"
server_key_version: "encryptionkey@0"
I20260806 09:25:40.331238 7006 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "40e0f22a71424e07bed66412e6d24f19"
format_stamp: "Formatted at 2026-08-06 09:25:40 on dist-test-slave-h3ft"
server_key: "4306d4d44746e5f12dafdfa827c0536f"
server_key_iv: "88307b3f26b4221c0a35802d1a42c23c"
server_key_version: "encryptionkey@0"
I20260806 09:25:40.337302 7006 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:25:40.341589 7031 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:40.343327 7006 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:25:40.343523 7006 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "40e0f22a71424e07bed66412e6d24f19"
format_stamp: "Formatted at 2026-08-06 09:25:40 on dist-test-slave-h3ft"
server_key: "4306d4d44746e5f12dafdfa827c0536f"
server_key_iv: "88307b3f26b4221c0a35802d1a42c23c"
server_key_version: "encryptionkey@0"
I20260806 09:25:40.343807 7006 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:25:40.363250 7006 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:25:40.368084 7006 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:25:40.368418 7006 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:25:40.369534 7006 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:25:40.371048 7006 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:25:40.371156 7006 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:40.371315 7006 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:25:40.371387 7006 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:25:40.412317 7006 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:38851
I20260806 09:25:40.412446 7176 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:38851 every 8 connection(s)
I20260806 09:25:40.413784 7006 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:25:40.417795 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 7006
I20260806 09:25:40.418001 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:25:40.418500 16730 external_mini_cluster.cc:1428] Setting key 692cfefe6d6ccfdb0785f5820dea7945
Aug 06 09:25:40 dist-test-slave-h3ft krb5kdc[6525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008340, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:25:40.439333 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.417089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:50699 (local address 127.16.86.190:34327)
0806 09:25:40.417722 (+ 633us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:40.417730 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:40.419300 (+ 1570us) server_negotiation.cc:407] Connection header received
0806 09:25:40.421858 (+ 2558us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:40.421862 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:40.421988 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:40.422137 (+ 149us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:40.424058 (+ 1921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.424904 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.426630 (+ 1726us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.427201 (+ 571us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.427323 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.431628 (+ 4305us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:40.431633 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:40.431637 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:40.431678 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:40.435598 (+ 3920us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:40.436616 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:40.436622 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:40.436626 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:40.436773 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:40.437275 (+ 502us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:40.437278 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:40.437281 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:40.437614 (+ 333us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:40.437812 (+ 198us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:40.438737 (+ 925us) server_negotiation.cc:299] Negotiation successful
0806 09:25:40.438922 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":461,"thread_start_us":282,"threads_started":1}
I20260806 09:25:40.440642 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.417908 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:50699)
0806 09:25:40.419061 (+ 1153us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:40.419120 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:40.421498 (+ 2378us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:40.422421 (+ 923us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:40.422430 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:40.423158 (+ 728us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:25:40.423831 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:40.423849 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.425052 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.425056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:40.426410 (+ 1354us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:40.426423 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.427462 (+ 1039us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.427467 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:40.427715 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.428594 (+ 879us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:25:40.428627 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:25:40.431402 (+ 2775us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:25:40.435789 (+ 4387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:40.435799 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:40.435854 (+ 55us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:40.436437 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:40.436898 (+ 461us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:25:40.436909 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:25:40.436911 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:25:40.437148 (+ 237us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:25:40.437774 (+ 626us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:25:40.437782 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:25:40.438283 (+ 501us) client_negotiation.cc:769] Sending connection context
0806 09:25:40.438603 (+ 320us) client_negotiation.cc:240] Negotiation successful
0806 09:25:40.439216 (+ 613us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":754,"thread_start_us":465,"threads_started":1}
I20260806 09:25:40.442570 7177 heartbeater.cc:347] Connected to a master server at 127.16.86.190:34327
I20260806 09:25:40.442994 7177 heartbeater.cc:464] Registering TS with master...
I20260806 09:25:40.444092 7177 heartbeater.cc:511] Master 127.16.86.190:34327 requested a full tablet report, sending...
I20260806 09:25:40.446178 6578 ts_manager.cc:194] Registered new tserver with Master: 40e0f22a71424e07bed66412e6d24f19 (127.16.86.131:38851)
I20260806 09:25:40.447606 6578 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:50699
I20260806 09:25:40.448721 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:25:40.470480 7179 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.454586 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58762 (local address 127.16.86.190:34327)
0806 09:25:40.454848 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:40.454858 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:40.455382 (+ 524us) server_negotiation.cc:407] Connection header received
0806 09:25:40.455635 (+ 253us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:40.455647 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:40.455723 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:40.455844 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:40.457627 (+ 1783us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.458513 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.460183 (+ 1670us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.460653 (+ 470us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.460804 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.463870 (+ 3066us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:40.463875 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:40.463879 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:40.463924 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:40.467248 (+ 3324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:40.468071 (+ 823us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:40.468076 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:40.468079 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:40.468227 (+ 148us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:40.468690 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:40.468694 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:40.468696 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:40.469100 (+ 404us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:40.469428 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:40.469779 (+ 351us) server_negotiation.cc:299] Negotiation successful
0806 09:25:40.470073 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20260806 09:25:40.475952 6574 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58762:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:25:40.483296 6574 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:25:40.521895 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.497473 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57302 (local address 127.16.86.129:43931)
0806 09:25:40.510922 (+ 13449us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:40.510930 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:40.510959 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:25:40.511120 (+ 161us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:40.511130 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:40.511462 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:40.511742 (+ 280us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:40.513830 (+ 2088us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.515670 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.519087 (+ 3417us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.520284 (+ 1197us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.520433 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.520701 (+ 268us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:40.521317 (+ 616us) server_negotiation.cc:299] Negotiation successful
0806 09:25:40.521508 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13263,"thread_start_us":13063,"threads_started":1}
I20260806 09:25:40.522588 7197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.497408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:41903 (local address 127.0.0.1:49032)
0806 09:25:40.510682 (+ 13274us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:40.510704 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:40.510865 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:40.512009 (+ 1144us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:40.512021 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:40.512074 (+ 53us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:40.513865 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:40.513880 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.515517 (+ 1637us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.515528 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:40.518645 (+ 3117us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:40.518662 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.520667 (+ 2005us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.520673 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:40.520956 (+ 283us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.521001 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 09:25:40.521204 (+ 203us) client_negotiation.cc:240] Negotiation successful
0806 09:25:40.521469 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13139,"mutex_wait_us":408,"thread_start_us":10947,"threads_started":1}
I20260806 09:25:40.523128 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.497678 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49032 (local address 127.16.86.130:41903)
0806 09:25:40.510269 (+ 12591us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:40.510285 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:40.510923 (+ 638us) server_negotiation.cc:407] Connection header received
0806 09:25:40.511121 (+ 198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:40.511130 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:40.511462 (+ 332us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:40.511725 (+ 263us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:40.514070 (+ 2345us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.515300 (+ 1230us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.518865 (+ 3565us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.520449 (+ 1584us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.520626 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.520868 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:40.522223 (+ 1355us) server_negotiation.cc:299] Negotiation successful
0806 09:25:40.522444 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12343,"thread_start_us":12184,"threads_started":1}
I20260806 09:25:40.523149 7198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.497218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:43931 (local address 127.0.0.1:57302)
0806 09:25:40.507760 (+ 10542us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:40.507823 (+ 63us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:40.508092 (+ 269us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:40.512664 (+ 4572us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:40.512670 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:40.512699 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:40.513487 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:40.513506 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.515916 (+ 2410us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.515924 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:40.518846 (+ 2922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:40.518857 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.520854 (+ 1997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.520861 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:40.521061 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.521103 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 09:25:40.522434 (+ 1331us) client_negotiation.cc:240] Negotiation successful
0806 09:25:40.522607 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10152,"mutex_wait_us":68,"thread_start_us":12361,"threads_started":1}
I20260806 09:25:40.523590 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.510548 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40790 (local address 127.16.86.131:38851)
0806 09:25:40.514418 (+ 3870us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:40.514427 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:40.514464 (+ 37us) server_negotiation.cc:407] Connection header received
0806 09:25:40.514585 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:40.514594 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:40.514862 (+ 268us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:40.515147 (+ 285us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:40.515909 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.517312 (+ 1403us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.520081 (+ 2769us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.521657 (+ 1576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.521794 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.522051 (+ 257us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:40.522819 (+ 768us) server_negotiation.cc:299] Negotiation successful
0806 09:25:40.523172 (+ 353us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3643,"thread_start_us":2840,"threads_started":1}
I20260806 09:25:40.523968 7203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.511191 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:38851 (local address 127.0.0.1:40790)
0806 09:25:40.512674 (+ 1483us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:40.512695 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:40.512838 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:40.515127 (+ 2289us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:40.515139 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:40.515172 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:40.515648 (+ 476us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:40.515663 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.517471 (+ 1808us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.517479 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:40.519849 (+ 2370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:40.519866 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.522221 (+ 2355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.522227 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:40.522417 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.522452 (+ 35us) client_negotiation.cc:769] Sending connection context
0806 09:25:40.522539 (+ 87us) client_negotiation.cc:240] Negotiation successful
0806 09:25:40.522669 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1320,"thread_start_us":1048,"threads_started":1}
I20260806 09:25:40.527865 7094 tablet_service.cc:1467] Processing CreateTablet for tablet 89366b44143f4b7d93f693bfe595dff0 (DEFAULT_TABLE table=test-table [id=e3afadc12098420ba16f269709a2bc7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:25:40.527993 6739 tablet_service.cc:1467] Processing CreateTablet for tablet 89366b44143f4b7d93f693bfe595dff0 (DEFAULT_TABLE table=test-table [id=e3afadc12098420ba16f269709a2bc7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:25:40.527992 6912 tablet_service.cc:1467] Processing CreateTablet for tablet 89366b44143f4b7d93f693bfe595dff0 (DEFAULT_TABLE table=test-table [id=e3afadc12098420ba16f269709a2bc7e]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:25:40.529299 7094 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89366b44143f4b7d93f693bfe595dff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:40.529711 6739 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89366b44143f4b7d93f693bfe595dff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:40.529726 6912 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 89366b44143f4b7d93f693bfe595dff0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:25:40.541977 7204 tablet_bootstrap.cc:492] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19: Bootstrap starting.
I20260806 09:25:40.545351 7204 tablet_bootstrap.cc:654] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:40.546600 7204 log.cc:824] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:40.548123 7205 tablet_bootstrap.cc:492] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252: Bootstrap starting.
I20260806 09:25:40.549007 7206 tablet_bootstrap.cc:492] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4: Bootstrap starting.
I20260806 09:25:40.555137 7205 tablet_bootstrap.cc:654] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:40.556456 7205 log.cc:824] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:40.557655 7206 tablet_bootstrap.cc:654] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4: Neither blocks nor log segments found. Creating new log.
I20260806 09:25:40.558759 7204 tablet_bootstrap.cc:492] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19: No bootstrap required, opened a new log
I20260806 09:25:40.559110 7204 ts_tablet_manager.cc:1397] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19: Time spent bootstrapping tablet: real 0.017s user 0.014s sys 0.002s
I20260806 09:25:40.559343 7206 log.cc:824] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4: Log is configured to *not* fsync() on all Append() calls
I20260806 09:25:40.561537 7205 tablet_bootstrap.cc:492] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252: No bootstrap required, opened a new log
I20260806 09:25:40.561996 7205 ts_tablet_manager.cc:1397] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252: Time spent bootstrapping tablet: real 0.015s user 0.000s sys 0.008s
I20260806 09:25:40.566116 7206 tablet_bootstrap.cc:492] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4: No bootstrap required, opened a new log
I20260806 09:25:40.566437 7206 ts_tablet_manager.cc:1397] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4: Time spent bootstrapping tablet: real 0.020s user 0.007s sys 0.004s
I20260806 09:25:40.569334 7204 raft_consensus.cc:348] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } }
I20260806 09:25:40.569809 7204 raft_consensus.cc:374] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:40.569914 7204 raft_consensus.cc:729] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e0f22a71424e07bed66412e6d24f19, State: Initialized, Role: FOLLOWER
I20260806 09:25:40.570585 7204 consensus_queue.cc:260] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } }
I20260806 09:25:40.571022 7205 raft_consensus.cc:348] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } }
I20260806 09:25:40.571605 7205 raft_consensus.cc:374] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:40.571723 7205 raft_consensus.cc:729] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398206deeb46492e87d93151bcf0b252, State: Initialized, Role: FOLLOWER
I20260806 09:25:40.572253 7177 heartbeater.cc:503] Master 127.16.86.190:34327 was elected leader, sending a full tablet report...
I20260806 09:25:40.572700 7204 ts_tablet_manager.cc:1428] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19: Time spent starting tablet: real 0.013s user 0.011s sys 0.001s
I20260806 09:25:40.572531 7205 consensus_queue.cc:260] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } }
I20260806 09:25:40.574168 6995 heartbeater.cc:503] Master 127.16.86.190:34327 was elected leader, sending a full tablet report...
I20260806 09:25:40.574826 7205 ts_tablet_manager.cc:1428] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252: Time spent starting tablet: real 0.013s user 0.004s sys 0.010s
I20260806 09:25:40.578415 7206 raft_consensus.cc:348] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } }
I20260806 09:25:40.578933 7206 raft_consensus.cc:374] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:25:40.579080 7206 raft_consensus.cc:729] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6da66d904e45219f37ff127f9dc2e4, State: Initialized, Role: FOLLOWER
I20260806 09:25:40.579769 7206 consensus_queue.cc:260] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } }
I20260806 09:25:40.581763 7206 ts_tablet_manager.cc:1428] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4: Time spent starting tablet: real 0.015s user 0.004s sys 0.011s
I20260806 09:25:40.584905 7212 raft_consensus.cc:482] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:25:40.585291 7212 raft_consensus.cc:504] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } }
I20260806 09:25:40.587407 7212 leader_election.cc:288] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d6da66d904e45219f37ff127f9dc2e4 (127.16.86.129:43931), 398206deeb46492e87d93151bcf0b252 (127.16.86.130:41903)
I20260806 09:25:40.594110 7181 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.587782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:41903 (local address 127.16.86.131:45301)
0806 09:25:40.588014 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:40.588036 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:40.588195 (+ 159us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:40.588624 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:40.588630 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:40.588670 (+ 40us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:40.588917 (+ 247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:40.588928 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.590192 (+ 1264us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.590204 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:40.592139 (+ 1935us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:40.592156 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.593392 (+ 1236us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.593400 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:40.593641 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.593692 (+ 51us) client_negotiation.cc:769] Sending connection context
0806 09:25:40.593777 (+ 85us) client_negotiation.cc:240] Negotiation successful
0806 09:25:40.593877 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":71}
I20260806 09:25:40.594295 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.588100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:45301 (local address 127.16.86.130:41903)
0806 09:25:40.588301 (+ 201us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:40.588307 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:40.588327 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:25:40.588399 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:40.588403 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:40.588469 (+ 66us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:40.588587 (+ 118us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:40.589062 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.590003 (+ 941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.592315 (+ 2312us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.593207 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.593356 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.593420 (+ 64us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:40.593838 (+ 418us) server_negotiation.cc:299] Negotiation successful
0806 09:25:40.593990 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:25:40.595580 6938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89366b44143f4b7d93f693bfe595dff0" candidate_uuid: "40e0f22a71424e07bed66412e6d24f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398206deeb46492e87d93151bcf0b252" is_pre_election: true
I20260806 09:25:40.596130 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.587756 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:44361 (local address 127.16.86.129:43931)
0806 09:25:40.588020 (+ 264us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:40.588030 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:40.589057 (+ 1027us) server_negotiation.cc:407] Connection header received
0806 09:25:40.589281 (+ 224us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:40.589285 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:40.589352 (+ 67us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:40.589462 (+ 110us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:25:40.590560 (+ 1098us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.591607 (+ 1047us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.593667 (+ 2060us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.595073 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.595213 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.595269 (+ 56us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:40.595695 (+ 426us) server_negotiation.cc:299] Negotiation successful
0806 09:25:40.595802 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:25:40.596282 7216 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.587800 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:43931 (local address 127.16.86.131:44361)
0806 09:25:40.588928 (+ 1128us) negotiation.cc:107] Waiting for socket to connect
0806 09:25:40.588944 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:25:40.589113 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:25:40.589759 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:25:40.589766 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:25:40.589798 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:25:40.590301 (+ 503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:40.590322 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.591778 (+ 1456us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.591785 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:40.593495 (+ 1710us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:25:40.593509 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.595302 (+ 1793us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.595308 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:25:40.595498 (+ 190us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.595537 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 09:25:40.595673 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 09:25:40.595833 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":984,"mutex_wait_us":246,"thread_start_us":459,"threads_started":1}
I20260806 09:25:40.596299 6938 raft_consensus.cc:2454] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40e0f22a71424e07bed66412e6d24f19 in term 0.
I20260806 09:25:40.597183 7036 leader_election.cc:302] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398206deeb46492e87d93151bcf0b252, 40e0f22a71424e07bed66412e6d24f19; no voters:
I20260806 09:25:40.597337 6763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89366b44143f4b7d93f693bfe595dff0" candidate_uuid: "40e0f22a71424e07bed66412e6d24f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d6da66d904e45219f37ff127f9dc2e4" is_pre_election: true
I20260806 09:25:40.597680 7212 raft_consensus.cc:2781] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:25:40.597817 7212 raft_consensus.cc:482] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:25:40.597915 7212 raft_consensus.cc:3037] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:40.597963 6763 raft_consensus.cc:2454] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 40e0f22a71424e07bed66412e6d24f19 in term 0.
I20260806 09:25:40.600652 7212 raft_consensus.cc:504] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } }
I20260806 09:25:40.601423 7212 leader_election.cc:288] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [CANDIDATE]: Term 1 election: Requested vote from peers 4d6da66d904e45219f37ff127f9dc2e4 (127.16.86.129:43931), 398206deeb46492e87d93151bcf0b252 (127.16.86.130:41903)
I20260806 09:25:40.601925 6763 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89366b44143f4b7d93f693bfe595dff0" candidate_uuid: "40e0f22a71424e07bed66412e6d24f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d6da66d904e45219f37ff127f9dc2e4"
I20260806 09:25:40.602039 6938 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "89366b44143f4b7d93f693bfe595dff0" candidate_uuid: "40e0f22a71424e07bed66412e6d24f19" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "398206deeb46492e87d93151bcf0b252"
I20260806 09:25:40.602212 6763 raft_consensus.cc:3037] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:40.602248 6938 raft_consensus.cc:3037] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:25:40.606109 6763 raft_consensus.cc:2454] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e0f22a71424e07bed66412e6d24f19 in term 1.
I20260806 09:25:40.606097 6938 raft_consensus.cc:2454] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 40e0f22a71424e07bed66412e6d24f19 in term 1.
I20260806 09:25:40.606619 7032 leader_election.cc:302] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 40e0f22a71424e07bed66412e6d24f19, 4d6da66d904e45219f37ff127f9dc2e4; no voters:
I20260806 09:25:40.607151 7212 raft_consensus.cc:2781] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:25:40.607864 7212 raft_consensus.cc:686] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 1 LEADER]: Becoming Leader. State: Replica: 40e0f22a71424e07bed66412e6d24f19, State: Running, Role: LEADER
I20260806 09:25:40.608464 7212 consensus_queue.cc:237] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } }
W20260806 09:25:40.613348 6825 tablet.cc:2369] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:25:40.614775 6574 catalog_manager.cc:5495] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 reported cstate change: term changed from 0 to 1, leader changed from <none> to 40e0f22a71424e07bed66412e6d24f19 (127.16.86.131). New cstate: current_term: 1 leader_uuid: "40e0f22a71424e07bed66412e6d24f19" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } health_report { overall_health: HEALTHY } } }
I20260806 09:25:40.657630 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.650847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40796 (local address 127.16.86.131:38851)
0806 09:25:40.651158 (+ 311us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:40.651167 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:40.651338 (+ 171us) server_negotiation.cc:407] Connection header received
0806 09:25:40.651691 (+ 353us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:40.651717 (+ 26us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:40.651818 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:40.652002 (+ 184us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:25:40.652445 (+ 443us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.653253 (+ 808us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.655048 (+ 1795us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.655516 (+ 468us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.655682 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.656323 (+ 641us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:25:40.656921 (+ 598us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:25:40.657108 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:40.657199 (+ 91us) server_negotiation.cc:299] Negotiation successful
0806 09:25:40.657296 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
W20260806 09:25:40.665777 7178 tablet.cc:2369] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:25:40.667856 6763 raft_consensus.cc:1261] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4 [term 1 FOLLOWER]: Refusing update from remote peer 40e0f22a71424e07bed66412e6d24f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:25:40.668005 6938 raft_consensus.cc:1261] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252 [term 1 FOLLOWER]: Refusing update from remote peer 40e0f22a71424e07bed66412e6d24f19: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:25:40.669140 7217 consensus_queue.cc:1035] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:25:40.669612 7212 consensus_queue.cc:1035] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:25:40.686517 7222 mvcc.cc:205] Tried to move back new op lower bound from 7315490163362443264 to 7315490163137847296. Current Snapshot: MvccSnapshot[applied={T|T < 7315490163362443264}]
I20260806 09:25:40.704365 6574 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:25:40.704720 6574 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58762:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:25:40.708992 6574 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Sending DeleteTablet for 3 replicas of tablet 89366b44143f4b7d93f693bfe595dff0
I20260806 09:25:40.710623 6912 tablet_service.cc:1514] Processing DeleteTablet for tablet 89366b44143f4b7d93f693bfe595dff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:25:40 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:49032
I20260806 09:25:40.710642 7094 tablet_service.cc:1514] Processing DeleteTablet for tablet 89366b44143f4b7d93f693bfe595dff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:25:40 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:40790
I20260806 09:25:40.710762 6739 tablet_service.cc:1514] Processing DeleteTablet for tablet 89366b44143f4b7d93f693bfe595dff0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:25:40 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:57302
I20260806 09:25:40.712185 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 6657
I20260806 09:25:40.721475 7237 tablet_replica.cc:330] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252: stopping tablet replica
I20260806 09:25:40.722836 7237 raft_consensus.cc:2229] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:25:40.724203 7237 raft_consensus.cc:2258] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:25:40.727689 7237 ts_tablet_manager.cc:1905] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:25:40.732878 7240 tablet_replica.cc:330] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4: stopping tablet replica
I20260806 09:25:40.733378 7240 raft_consensus.cc:2229] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:25:40.733956 7240 raft_consensus.cc:2258] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:25:40.734884 7236 tablet_replica.cc:330] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19: stopping tablet replica
I20260806 09:25:40.735478 7236 raft_consensus.cc:2229] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:25:40.736315 7236 raft_consensus.cc:2258] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:25:40.739089 7240 ts_tablet_manager.cc:1905] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4: Deleting tablet data with delete state TABLET_DATA_DELETED
Aug 06 09:25:40 dist-test-slave-h3ft krb5kdc[6525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:25:40.741146 7236 ts_tablet_manager.cc:1905] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:25:40.741534 7237 ts_tablet_manager.cc:1918] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:25:40.741914 7237 log.cc:1196] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/89366b44143f4b7d93f693bfe595dff0
I20260806 09:25:40.742672 7237 ts_tablet_manager.cc:1939] T 89366b44143f4b7d93f693bfe595dff0 P 398206deeb46492e87d93151bcf0b252: Deleting consensus metadata
I20260806 09:25:40.745539 6561 catalog_manager.cc:4841] TS 398206deeb46492e87d93151bcf0b252 (127.16.86.130:41903): tablet 89366b44143f4b7d93f693bfe595dff0 (table test-table [id=e3afadc12098420ba16f269709a2bc7e]) successfully deleted
I20260806 09:25:40.751753 7240 ts_tablet_manager.cc:1918] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:25:40.752427 7240 log.cc:1196] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/89366b44143f4b7d93f693bfe595dff0
I20260806 09:25:40.753103 7240 ts_tablet_manager.cc:1939] T 89366b44143f4b7d93f693bfe595dff0 P 4d6da66d904e45219f37ff127f9dc2e4: Deleting consensus metadata
I20260806 09:25:40.753499 7236 ts_tablet_manager.cc:1918] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:25:40.753746 7236 log.cc:1196] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/89366b44143f4b7d93f693bfe595dff0
I20260806 09:25:40.754298 7236 ts_tablet_manager.cc:1939] T 89366b44143f4b7d93f693bfe595dff0 P 40e0f22a71424e07bed66412e6d24f19: Deleting consensus metadata
I20260806 09:25:40.752277 7199 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.713843 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57318 (local address 127.16.86.129:43931)
0806 09:25:40.714126 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:40.714135 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:40.726594 (+ 12459us) server_negotiation.cc:407] Connection header received
0806 09:25:40.727308 (+ 714us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:40.727317 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:40.727452 (+ 135us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:40.727559 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:40.730680 (+ 3121us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.732591 (+ 1911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.734678 (+ 2087us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.735344 (+ 666us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.735497 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.742291 (+ 6794us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:40.742315 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:40.742342 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:40.742406 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:40.747160 (+ 4754us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:40.748368 (+ 1208us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:40.748379 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:40.748383 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:40.748686 (+ 303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:40.749257 (+ 571us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:40.749263 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:40.749267 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:40.749919 (+ 652us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:40.750289 (+ 370us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:40.751129 (+ 840us) server_negotiation.cc:299] Negotiation successful
0806 09:25:40.751455 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":83}
I20260806 09:25:40.755300 6558 catalog_manager.cc:4841] TS 4d6da66d904e45219f37ff127f9dc2e4 (127.16.86.129:43931): tablet 89366b44143f4b7d93f693bfe595dff0 (table test-table [id=e3afadc12098420ba16f269709a2bc7e]) successfully deleted
I20260806 09:25:40.756337 6558 catalog_manager.cc:4841] TS 40e0f22a71424e07bed66412e6d24f19 (127.16.86.131:38851): tablet 89366b44143f4b7d93f693bfe595dff0 (table test-table [id=e3afadc12098420ba16f269709a2bc7e]) successfully deleted
I20260806 09:25:40.756389 6816 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:40.861259 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 6657
I20260806 09:25:40.889463 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 6832
Aug 06 09:25:40 dist-test-slave-h3ft krb5kdc[6525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:25:40.908932 7200 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:40.890510 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49046 (local address 127.16.86.130:41903)
0806 09:25:40.890802 (+ 292us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:40.890810 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:40.890839 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:25:40.890923 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:40.890929 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:40.891061 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:40.891199 (+ 138us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:40.892697 (+ 1498us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.893899 (+ 1202us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.895546 (+ 1647us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:40.896189 (+ 643us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:40.896333 (+ 144us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:40.901562 (+ 5229us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:40.901579 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:40.901599 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:40.901654 (+ 55us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:40.905721 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:40.906478 (+ 757us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:40.906488 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:40.906492 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:40.906662 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:40.907076 (+ 414us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:40.907089 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:40.907094 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:40.907623 (+ 529us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:40.907935 (+ 312us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:40.908281 (+ 346us) server_negotiation.cc:299] Negotiation successful
0806 09:25:40.908522 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 09:25:40.910020 6989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:41.011102 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 6832
I20260806 09:25:41.039222 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 7006
Aug 06 09:25:41 dist-test-slave-h3ft krb5kdc[6525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008337, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:25:41.060302 7201 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:25:41.040333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40798 (local address 127.16.86.131:38851)
0806 09:25:41.040631 (+ 298us) server_negotiation.cc:206] Beginning negotiation
0806 09:25:41.040639 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:25:41.040710 (+ 71us) server_negotiation.cc:407] Connection header received
0806 09:25:41.040812 (+ 102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:25:41.040819 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:25:41.040926 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:25:41.041061 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:25:41.042471 (+ 1410us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:41.043958 (+ 1487us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:41.045511 (+ 1553us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:25:41.046236 (+ 725us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:25:41.046427 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:25:41.052727 (+ 6300us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:25:41.052749 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:25:41.052767 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:25:41.052838 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:25:41.056466 (+ 3628us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:41.057651 (+ 1185us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:41.057659 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:41.057661 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:41.057855 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:25:41.058401 (+ 546us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:25:41.058406 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:25:41.058408 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:25:41.058896 (+ 488us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:25:41.059263 (+ 367us) server_negotiation.cc:1035] Waiting for connection context
0806 09:25:41.059738 (+ 475us) server_negotiation.cc:299] Negotiation successful
0806 09:25:41.059950 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:25:41.061951 7170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:41.193812 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 7006
I20260806 09:25:41.226454 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 6539
I20260806 09:25:41.228003 6618 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:25:41.342358 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 6539
I20260806 09:26:01.380867 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:34327
--webserver_interface=127.16.86.190
--webserver_port=39097
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:34327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:26:01.830757 7248 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:01.831503 7248 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:01.831656 7248 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:01.841295 7248 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 09:26:01.841580 7248 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 09:26:01.841674 7248 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 09:26:01.841861 7248 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:26:01.841991 7248 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:01.842121 7248 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:01.842264 7248 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:26:01.842394 7248 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:26:01.858239 7248 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:34327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=39097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:01.860214 7248 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:01.862707 7248 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:01.899580 7258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:01.899544 7255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:01.899938 7256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:01.901083 7248 server_base.cc:1034] running on GCE node
I20260806 09:26:01.902423 7248 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:01.904974 7248 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:01.906447 7248 hybrid_clock.cc:648] HybridClock initialized: now 1786008361906308 us; error 110 us; skew 500 ppm
I20260806 09:26:01.907186 7248 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:01 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008361, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:01.920096 7248 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:26:01.922938 7248 webserver.cc:460] Webserver started at http://127.16.86.190:39097/ using document root <none> and password file <none>
I20260806 09:26:01.924263 7248 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:01.924489 7248 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:01.933578 7248 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260806 09:26:01.938437 7268 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:01.940681 7248 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 09:26:01.941046 7248 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "e063cec8fc994c39af3bc0a6b3d75ffa"
format_stamp: "Formatted at 2026-08-06 09:25:38 on dist-test-slave-h3ft"
server_key: "a2cfafb68656cf53cdc4ac7d524d1c9f"
server_key_iv: "c464e578db4287f2116d86c96412b70a"
server_key_version: "encryptionkey@0"
I20260806 09:26:01.942588 7248 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:01.966446 7248 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:01.973992 7248 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:01.974541 7248 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:02.004279 7328 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:34327 every 8 connection(s)
I20260806 09:26:02.004297 7248 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:34327
I20260806 09:26:02.006937 7248 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:26:02.016254 7329 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:26:02.017169 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 7248
I20260806 09:26:02.018297 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:43931
--local_ip_for_outbound_sockets=127.16.86.129
--tserver_master_addrs=127.16.86.190:34327
--webserver_port=39767
--webserver_interface=127.16.86.129
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:26:02.020900 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Bootstrap starting.
I20260806 09:26:02.048241 7329 log.cc:824] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:02.070116 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:26:02.070904 7329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Bootstrap complete.
I20260806 09:26:02.086284 7329 raft_consensus.cc:348] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } }
I20260806 09:26:02.087702 7329 raft_consensus.cc:729] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063cec8fc994c39af3bc0a6b3d75ffa, State: Initialized, Role: FOLLOWER
I20260806 09:26:02.088598 7329 consensus_queue.cc:260] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } }
I20260806 09:26:02.088989 7329 raft_consensus.cc:388] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:26:02.089165 7329 raft_consensus.cc:482] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:26:02.089859 7329 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 1 FOLLOWER]: Advancing to term 2
I20260806 09:26:02.097144 7329 raft_consensus.cc:504] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } }
I20260806 09:26:02.098198 7329 leader_election.cc:302] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e063cec8fc994c39af3bc0a6b3d75ffa; no voters:
I20260806 09:26:02.099289 7329 leader_election.cc:288] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [CANDIDATE]: Term 2 election: Requested vote from peers
I20260806 09:26:02.099690 7333 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 2 FOLLOWER]: Leader election won for term 2
I20260806 09:26:02.101501 7333 raft_consensus.cc:686] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 2 LEADER]: Becoming Leader. State: Replica: e063cec8fc994c39af3bc0a6b3d75ffa, State: Running, Role: LEADER
I20260806 09:26:02.102571 7333 consensus_queue.cc:237] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } }
I20260806 09:26:02.103166 7329 sys_catalog.cc:564] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:26:02.106642 7334 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } } }
I20260806 09:26:02.107810 7334 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: This master's current role is: LEADER
I20260806 09:26:02.106601 7335 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e063cec8fc994c39af3bc0a6b3d75ffa. Latest consensus state: current_term: 2 leader_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } } }
I20260806 09:26:02.108929 7335 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: This master's current role is: LEADER
I20260806 09:26:02.111500 7341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:26:02.123555 7341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:26:02.126118 7341 catalog_manager.cc:1259] Loaded cluster ID: 6e87a7edfa5a4e62865e50dd6680ff96
I20260806 09:26:02.126371 7341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:26:02.132112 7341 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:26:02.136376 7341 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Loaded TSK: 0
I20260806 09:26:02.188254 7341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Generated new TSK 1
I20260806 09:26:02.188922 7341 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:26:02.517722 7331 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:02.518187 7331 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:02.518383 7331 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:02.526930 7331 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:02.527179 7331 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:02.527397 7331 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:26:02.539467 7331 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:43931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=39767
--tserver_master_addrs=127.16.86.190:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:02.541330 7331 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:02.543052 7331 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:02.562338 7362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:02.562119 7363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:02.562096 7365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:02.563128 7331 server_base.cc:1034] running on GCE node
I20260806 09:26:02.564105 7331 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:02.565575 7331 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:02.566962 7331 hybrid_clock.cc:648] HybridClock initialized: now 1786008362566841 us; error 109 us; skew 500 ppm
I20260806 09:26:02.567548 7331 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:02 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008362, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:02.572274 7331 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:26:02.574483 7331 webserver.cc:460] Webserver started at http://127.16.86.129:39767/ using document root <none> and password file <none>
I20260806 09:26:02.575644 7331 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:02.575822 7331 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:02.584451 7331 fs_manager.cc:709] Time spent opening directory manager: real 0.006s user 0.001s sys 0.006s
I20260806 09:26:02.588960 7373 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:02.591445 7331 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260806 09:26:02.591697 7331 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "4d6da66d904e45219f37ff127f9dc2e4"
format_stamp: "Formatted at 2026-08-06 09:25:39 on dist-test-slave-h3ft"
server_key: "3bb55387eae8f12ce29f3bfb0897e06b"
server_key_iv: "02b7e1de7c82950ee2880d78645e3666"
server_key_version: "encryptionkey@0"
I20260806 09:26:02.593077 7331 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:02.640050 7331 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:02.646138 7331 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:02.646487 7331 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:02.647920 7331 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:02.649995 7331 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:02.650115 7331 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:02.650209 7331 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:02.650272 7331 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:02.696590 7331 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:43931
I20260806 09:26:02.696838 7524 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:43931 every 8 connection(s)
I20260806 09:26:02.698601 7331 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:26:02.700834 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 7331
I20260806 09:26:02.701781 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:41903
--local_ip_for_outbound_sockets=127.16.86.130
--tserver_master_addrs=127.16.86.190:34327
--webserver_port=45413
--webserver_interface=127.16.86.130
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:26:02 dist-test-slave-h3ft krb5kdc[6525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008362, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:26:02.732918 7527 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:02.703203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:46133 (local address 127.16.86.190:34327)
0806 09:26:02.704067 (+ 864us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:02.704089 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:02.705262 (+ 1173us) server_negotiation.cc:407] Connection header received
0806 09:26:02.708028 (+ 2766us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:02.708057 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:02.708638 (+ 581us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:02.709184 (+ 546us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:02.711513 (+ 2329us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:02.713748 (+ 2235us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:02.716026 (+ 2278us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:02.716836 (+ 810us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:02.717015 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:02.722874 (+ 5859us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:02.722884 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:02.722905 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:02.722958 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:02.726450 (+ 3492us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:02.728022 (+ 1572us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:02.728030 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:02.728039 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:02.728299 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:02.729011 (+ 712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:02.729015 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:02.729017 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:02.729626 (+ 609us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:02.729821 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:02.731374 (+ 1553us) server_negotiation.cc:299] Negotiation successful
0806 09:26:02.731805 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":543,"thread_start_us":347,"threads_started":1}
I20260806 09:26:02.734015 7529 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:02.703901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:46133)
0806 09:26:02.705002 (+ 1101us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:02.705061 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:02.707236 (+ 2175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:02.709538 (+ 2302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:02.709549 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:02.710293 (+ 744us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:02.711211 (+ 918us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:02.711238 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:02.714058 (+ 2820us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:02.714063 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:02.715730 (+ 1667us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:02.715749 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:02.717161 (+ 1412us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:02.717171 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:02.717556 (+ 385us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:02.718593 (+ 1037us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:26:02.718634 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:02.722437 (+ 3803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:02.726697 (+ 4260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:02.726713 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:02.726741 (+ 28us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:02.727769 (+ 1028us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:02.728456 (+ 687us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:02.728464 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:02.728470 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:02.728823 (+ 353us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:02.729812 (+ 989us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:02.729825 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:02.730479 (+ 654us) client_negotiation.cc:769] Sending connection context
0806 09:26:02.730968 (+ 489us) client_negotiation.cc:240] Negotiation successful
0806 09:26:02.731826 (+ 858us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":745,"thread_start_us":431,"threads_started":1}
I20260806 09:26:02.737622 7525 heartbeater.cc:347] Connected to a master server at 127.16.86.190:34327
I20260806 09:26:02.738024 7525 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:02.738911 7525 heartbeater.cc:511] Master 127.16.86.190:34327 requested a full tablet report, sending...
I20260806 09:26:02.741694 7286 ts_manager.cc:194] Registered new tserver with Master: 4d6da66d904e45219f37ff127f9dc2e4 (127.16.86.129:43931)
I20260806 09:26:02.743510 7286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:46133
W20260806 09:26:03.164419 7530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:03.164852 7530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:03.165030 7530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:03.174151 7530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:03.174368 7530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:03.174574 7530 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:26:03.187609 7530 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:41903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=45413
--tserver_master_addrs=127.16.86.190:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:03.189713 7530 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:03.191561 7530 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:03.209045 7544 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:03.209148 7542 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:03.209023 7538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:03.215544 7530 server_base.cc:1034] running on GCE node
I20260806 09:26:03.216604 7530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:03.217993 7530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:03.219278 7530 hybrid_clock.cc:648] HybridClock initialized: now 1786008363219222 us; error 55 us; skew 500 ppm
I20260806 09:26:03.219739 7530 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:03 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008363, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:03.224422 7530 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:26:03.226688 7530 webserver.cc:460] Webserver started at http://127.16.86.130:45413/ using document root <none> and password file <none>
I20260806 09:26:03.227766 7530 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:03.227981 7530 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:03.236261 7530 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260806 09:26:03.240504 7552 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:03.242626 7530 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260806 09:26:03.242937 7530 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "398206deeb46492e87d93151bcf0b252"
format_stamp: "Formatted at 2026-08-06 09:25:39 on dist-test-slave-h3ft"
server_key: "826cb9293b0c4a2ed89f6ac7614c4239"
server_key_iv: "7442e7b4e151955c9327a4e2ff1fda8f"
server_key_version: "encryptionkey@0"
I20260806 09:26:03.244380 7530 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:03.262161 7530 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:03.267858 7530 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:03.268227 7530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:03.269870 7530 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:03.271811 7530 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:03.271916 7530 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:03.272010 7530 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:03.272109 7530 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:03.319801 7530 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:41903
I20260806 09:26:03.319994 7704 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:41903 every 8 connection(s)
I20260806 09:26:03.321862 7530 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:26:03.331914 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 7530
I20260806 09:26:03.332662 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:38851
--local_ip_for_outbound_sockets=127.16.86.131
--tserver_master_addrs=127.16.86.190:34327
--webserver_port=38961
--webserver_interface=127.16.86.131
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Aug 06 09:26:03 dist-test-slave-h3ft krb5kdc[6525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008363, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:26:03.352188 7707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:03.325582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:60037 (local address 127.16.86.190:34327)
0806 09:26:03.326175 (+ 593us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:03.326180 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:03.327545 (+ 1365us) server_negotiation.cc:407] Connection header received
0806 09:26:03.329713 (+ 2168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:03.329719 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:03.329822 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:03.329954 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:03.332117 (+ 2163us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:03.333321 (+ 1204us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:03.336263 (+ 2942us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:03.337046 (+ 783us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:03.337220 (+ 174us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:03.342796 (+ 5576us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:03.342806 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:03.342814 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:03.342890 (+ 76us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:03.347619 (+ 4729us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:03.348726 (+ 1107us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:03.348734 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:03.348742 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:03.348949 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:03.349506 (+ 557us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:03.349511 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:03.349515 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:03.349916 (+ 401us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:03.350126 (+ 210us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:03.351444 (+ 1318us) server_negotiation.cc:299] Negotiation successful
0806 09:26:03.351658 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":302,"threads_started":1}
I20260806 09:26:03.354329 7709 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:03.326273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:60037)
0806 09:26:03.327332 (+ 1059us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:03.327392 (+ 60us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:03.329388 (+ 1996us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:03.330236 (+ 848us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:03.330245 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:03.330873 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:03.331702 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:03.331732 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:03.333568 (+ 1836us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:03.333576 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:03.336004 (+ 2428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:03.336027 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:03.337502 (+ 1475us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:03.337511 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:03.337860 (+ 349us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:03.338924 (+ 1064us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:26:03.338965 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:03.342351 (+ 3386us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:03.347891 (+ 5540us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:03.347902 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:03.347921 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:03.348554 (+ 633us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:03.349075 (+ 521us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:03.349080 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:03.349084 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:03.349368 (+ 284us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:03.350148 (+ 780us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:03.350163 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:03.350871 (+ 708us) client_negotiation.cc:769] Sending connection context
0806 09:26:03.351326 (+ 455us) client_negotiation.cc:240] Negotiation successful
0806 09:26:03.352141 (+ 815us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":629,"thread_start_us":389,"threads_started":1}
I20260806 09:26:03.356424 7705 heartbeater.cc:347] Connected to a master server at 127.16.86.190:34327
I20260806 09:26:03.356827 7705 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:03.357834 7705 heartbeater.cc:511] Master 127.16.86.190:34327 requested a full tablet report, sending...
I20260806 09:26:03.359753 7286 ts_manager.cc:194] Registered new tserver with Master: 398206deeb46492e87d93151bcf0b252 (127.16.86.130:41903)
I20260806 09:26:03.360587 7286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:60037
I20260806 09:26:03.748603 7525 heartbeater.cc:503] Master 127.16.86.190:34327 was elected leader, sending a full tablet report...
W20260806 09:26:03.798337 7710 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:03.798885 7710 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:03.799037 7710 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:03.808326 7710 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:03.808523 7710 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:03.808682 7710 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:26:03.821115 7710 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:38851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=38961
--tserver_master_addrs=127.16.86.190:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:03.822916 7710 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:03.824754 7710 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:03.843878 7719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:03.843849 7721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:03.843863 7717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:03.853808 7710 server_base.cc:1034] running on GCE node
I20260806 09:26:03.855871 7710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:03.858182 7710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:03.859543 7710 hybrid_clock.cc:648] HybridClock initialized: now 1786008363859476 us; error 65 us; skew 500 ppm
I20260806 09:26:03.860085 7710 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:03 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008363, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:03.865170 7710 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:26:03.868103 7710 webserver.cc:460] Webserver started at http://127.16.86.131:38961/ using document root <none> and password file <none>
I20260806 09:26:03.869215 7710 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:03.869436 7710 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:03.877640 7710 fs_manager.cc:709] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:26:03.882526 7734 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:03.885344 7710 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 09:26:03.885607 7710 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "40e0f22a71424e07bed66412e6d24f19"
format_stamp: "Formatted at 2026-08-06 09:25:40 on dist-test-slave-h3ft"
server_key: "4306d4d44746e5f12dafdfa827c0536f"
server_key_iv: "88307b3f26b4221c0a35802d1a42c23c"
server_key_version: "encryptionkey@0"
I20260806 09:26:03.886838 7710 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:03.920606 7710 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:03.926059 7710 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:03.926402 7710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:03.927757 7710 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:03.929580 7710 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:03.929740 7710 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:03.929899 7710 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:03.930001 7710 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:03.985328 7710 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:38851
I20260806 09:26:03.985462 7881 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:38851 every 8 connection(s)
I20260806 09:26:03.987380 7710 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:26:03.997366 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 7710
Aug 06 09:26:04 dist-test-slave-h3ft krb5kdc[6525](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008363, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:26:04.024478 7887 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:03.999192 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48998 (local address 127.16.86.190:34327)
0806 09:26:04.000583 (+ 1391us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:04.000587 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:04.000609 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:26:04.000669 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:04.000673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:04.000761 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:04.000885 (+ 124us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:04.003205 (+ 2320us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.005461 (+ 2256us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.008150 (+ 2689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.009006 (+ 856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.009196 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.013579 (+ 4383us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:04.013585 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:04.013589 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:04.013635 (+ 46us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:04.019240 (+ 5605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:04.020965 (+ 1725us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:04.020973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:04.020977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:04.021452 (+ 475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:04.022234 (+ 782us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:04.022240 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:04.022242 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:04.022786 (+ 544us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:04.023334 (+ 548us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:04.023769 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 09:26:04.024010 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":53,"server-negotiator.queue_time_us":1226,"spinlock_wait_cycles":768,"thread_start_us":994,"threads_started":1}
I20260806 09:26:04.025583 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:03.991399 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:59129 (local address 127.16.86.190:34327)
0806 09:26:03.992208 (+ 809us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:03.992216 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:03.994203 (+ 1987us) server_negotiation.cc:407] Connection header received
0806 09:26:03.996617 (+ 2414us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:03.996624 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:03.996761 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:03.996911 (+ 150us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:04.000255 (+ 3344us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.001656 (+ 1401us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.005530 (+ 3874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.006477 (+ 947us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.006672 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.013867 (+ 7195us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:04.013873 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:04.013880 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:04.013941 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:04.019207 (+ 5266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:04.020965 (+ 1758us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:04.020973 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:04.020977 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:04.021452 (+ 475us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:04.022428 (+ 976us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:04.022433 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:04.022436 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:04.022788 (+ 352us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:04.023044 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:04.024528 (+ 1484us) server_negotiation.cc:299] Negotiation successful
0806 09:26:04.025180 (+ 652us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"spinlock_wait_cycles":6912,"thread_start_us":430,"threads_started":1}
I20260806 09:26:04.027360 7885 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:03.992824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:59129)
0806 09:26:03.993896 (+ 1072us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:03.993947 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:03.996173 (+ 2226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:03.997452 (+ 1279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:03.997471 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:03.998594 (+ 1123us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:03.999846 (+ 1252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:03.999889 (+ 43us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.001967 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.001972 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:04.004200 (+ 2228us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:04.004877 (+ 677us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.006924 (+ 2047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.006934 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:04.007485 (+ 551us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.008635 (+ 1150us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:26:04.008678 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:04.013458 (+ 4780us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:04.019632 (+ 6174us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:04.019648 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:04.019677 (+ 29us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:04.020616 (+ 939us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:04.021685 (+ 1069us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:04.021693 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:04.021698 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:04.022244 (+ 546us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:04.023122 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:04.023139 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:04.023835 (+ 696us) client_negotiation.cc:769] Sending connection context
0806 09:26:04.024338 (+ 503us) client_negotiation.cc:240] Negotiation successful
0806 09:26:04.025454 (+ 1116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":721,"thread_start_us":434,"threads_started":1}
I20260806 09:26:04.029872 7882 heartbeater.cc:347] Connected to a master server at 127.16.86.190:34327
I20260806 09:26:04.030361 7882 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:04.031502 7882 heartbeater.cc:511] Master 127.16.86.190:34327 requested a full tablet report, sending...
I20260806 09:26:04.033784 7286 ts_manager.cc:194] Registered new tserver with Master: 40e0f22a71424e07bed66412e6d24f19 (127.16.86.131:38851)
I20260806 09:26:04.034785 7286 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:59129
I20260806 09:26:04.042399 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:26:04.051636 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.044213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49010 (local address 127.16.86.190:34327)
0806 09:26:04.044541 (+ 328us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:04.044549 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:04.044932 (+ 383us) server_negotiation.cc:407] Connection header received
0806 09:26:04.045236 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:04.045240 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:04.045409 (+ 169us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:04.045608 (+ 199us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:26:04.046298 (+ 690us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.047119 (+ 821us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.048879 (+ 1760us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.049431 (+ 552us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.049613 (+ 182us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.050519 (+ 906us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:26:04.050778 (+ 259us) server_negotiation.cc:394] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
0806 09:26:04.051292 (+ 514us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:49010: token expired
Metrics: {"server-negotiator.queue_time_us":119}
W20260806 09:26:04.051779 7884 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:49010: token expired
W20260806 09:26:04.053074 7187 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.16.86.190:34327)
I20260806 09:26:04.070789 7884 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.053938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49026 (local address 127.16.86.190:34327)
0806 09:26:04.054166 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:04.054173 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:04.054278 (+ 105us) server_negotiation.cc:407] Connection header received
0806 09:26:04.054495 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:04.054501 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:04.054585 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:04.054698 (+ 113us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:04.056344 (+ 1646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.057125 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.058625 (+ 1500us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.059122 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.059262 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.063465 (+ 4203us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:04.063473 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:04.063478 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:04.063550 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:04.067205 (+ 3655us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:04.068528 (+ 1323us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:04.068537 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:04.068545 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:04.068739 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:04.069160 (+ 421us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:04.069166 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:04.069169 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:04.069535 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:04.069848 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:04.070150 (+ 302us) server_negotiation.cc:299] Negotiation successful
0806 09:26:04.070418 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 09:26:04.075466 7286 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49026:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:26:04.083845 7286 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:26:04.128400 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.102125 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46058 (local address 127.16.86.130:41903)
0806 09:26:04.102850 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:04.102861 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:04.115587 (+ 12726us) server_negotiation.cc:407] Connection header received
0806 09:26:04.115882 (+ 295us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:04.115894 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:04.116409 (+ 515us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:04.116731 (+ 322us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:04.118953 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.120986 (+ 2033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.124866 (+ 3880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.126610 (+ 1744us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.126764 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.126985 (+ 221us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:04.127703 (+ 718us) server_negotiation.cc:299] Negotiation successful
0806 09:26:04.127944 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":529,"thread_start_us":374,"threads_started":1}
I20260806 09:26:04.128469 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.102951 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49550 (local address 127.16.86.131:38851)
0806 09:26:04.117371 (+ 14420us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:04.117385 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:04.117436 (+ 51us) server_negotiation.cc:407] Connection header received
0806 09:26:04.117586 (+ 150us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:04.117596 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:04.118003 (+ 407us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:04.118348 (+ 345us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:04.119544 (+ 1196us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.121402 (+ 1858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.124665 (+ 3263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.126505 (+ 1840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.126714 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.127073 (+ 359us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:04.127881 (+ 808us) server_negotiation.cc:299] Negotiation successful
0806 09:26:04.128066 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14123,"thread_start_us":13679,"threads_started":1}
I20260806 09:26:04.128485 7902 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.102958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:41903 (local address 127.0.0.1:46058)
0806 09:26:04.115280 (+ 12322us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:04.115322 (+ 42us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:04.115647 (+ 325us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:04.117803 (+ 2156us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:04.117815 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:04.117858 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:04.118494 (+ 636us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:04.118534 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.121209 (+ 2675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.121214 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:04.124574 (+ 3360us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:04.124597 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.126953 (+ 2356us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.126960 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:04.127292 (+ 332us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.127400 (+ 108us) client_negotiation.cc:769] Sending connection context
0806 09:26:04.127599 (+ 199us) client_negotiation.cc:240] Negotiation successful
0806 09:26:04.127884 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12078,"mutex_wait_us":10,"thread_start_us":12499,"threads_started":1}
I20260806 09:26:04.129504 7898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.102015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:38851 (local address 127.0.0.1:49550)
0806 09:26:04.103988 (+ 1973us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:04.104052 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:04.104359 (+ 307us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:04.118446 (+ 14087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:04.118457 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:04.118517 (+ 60us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:04.119183 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:04.119201 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.121648 (+ 2447us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.121655 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:04.124419 (+ 2764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:04.124441 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.127316 (+ 2875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.127322 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:04.127611 (+ 289us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.127664 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 09:26:04.128803 (+ 1139us) client_negotiation.cc:240] Negotiation successful
0806 09:26:04.129020 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1464,"mutex_wait_us":221,"thread_start_us":1262,"threads_started":1}
I20260806 09:26:04.129899 7899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.105010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:43931 (local address 127.0.0.1:49544)
0806 09:26:04.115815 (+ 10805us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:04.115862 (+ 47us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:04.116241 (+ 379us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:04.120479 (+ 4238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:04.120485 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:04.120522 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:04.120952 (+ 430us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:04.120965 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.123370 (+ 2405us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.123377 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:04.125667 (+ 2290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:04.125682 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.128423 (+ 2741us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.128433 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:04.128664 (+ 231us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.128701 (+ 37us) client_negotiation.cc:769] Sending connection context
0806 09:26:04.128783 (+ 82us) client_negotiation.cc:240] Negotiation successful
0806 09:26:04.129280 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10643,"mutex_wait_us":343,"thread_start_us":9744,"threads_started":1}
I20260806 09:26:04.131505 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.105433 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49544 (local address 127.16.86.129:43931)
0806 09:26:04.119609 (+ 14176us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:04.119621 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:04.119662 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:26:04.119797 (+ 135us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:04.119807 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:04.120166 (+ 359us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:04.120467 (+ 301us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:04.121223 (+ 756us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.123164 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.125895 (+ 2731us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.128087 (+ 2192us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.129893 (+ 1806us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.130465 (+ 572us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:04.130917 (+ 452us) server_negotiation.cc:299] Negotiation successful
0806 09:26:04.131213 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13916,"thread_start_us":13489,"threads_started":1}
I20260806 09:26:04.133646 7616 tablet_service.cc:1467] Processing CreateTablet for tablet fff2d7646f1448f68894bdde5691eee5 (DEFAULT_TABLE table=test-table [id=28d1c821cc8c43b3b51480875861e171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:04.135426 7616 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff2d7646f1448f68894bdde5691eee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:04.136914 7444 tablet_service.cc:1467] Processing CreateTablet for tablet fff2d7646f1448f68894bdde5691eee5 (DEFAULT_TABLE table=test-table [id=28d1c821cc8c43b3b51480875861e171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:04.139217 7444 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff2d7646f1448f68894bdde5691eee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:04.139209 7797 tablet_service.cc:1467] Processing CreateTablet for tablet fff2d7646f1448f68894bdde5691eee5 (DEFAULT_TABLE table=test-table [id=28d1c821cc8c43b3b51480875861e171]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:04.140930 7797 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet fff2d7646f1448f68894bdde5691eee5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:04.154462 7905 tablet_bootstrap.cc:492] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252: Bootstrap starting.
I20260806 09:26:04.158607 7905 tablet_bootstrap.cc:654] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:04.160006 7908 tablet_bootstrap.cc:492] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19: Bootstrap starting.
I20260806 09:26:04.160367 7905 log.cc:824] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:04.163365 7907 tablet_bootstrap.cc:492] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4: Bootstrap starting.
I20260806 09:26:04.165726 7905 tablet_bootstrap.cc:492] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252: No bootstrap required, opened a new log
I20260806 09:26:04.166049 7905 ts_tablet_manager.cc:1397] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.004s
I20260806 09:26:04.166184 7908 tablet_bootstrap.cc:654] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:04.167737 7908 log.cc:824] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:04.168161 7907 tablet_bootstrap.cc:654] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:04.169907 7907 log.cc:824] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:04.172282 7908 tablet_bootstrap.cc:492] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19: No bootstrap required, opened a new log
I20260806 09:26:04.172710 7908 ts_tablet_manager.cc:1397] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19: Time spent bootstrapping tablet: real 0.013s user 0.009s sys 0.000s
I20260806 09:26:04.174160 7907 tablet_bootstrap.cc:492] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4: No bootstrap required, opened a new log
I20260806 09:26:04.174518 7907 ts_tablet_manager.cc:1397] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.002s
I20260806 09:26:04.179494 7905 raft_consensus.cc:348] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } }
I20260806 09:26:04.180058 7905 raft_consensus.cc:374] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:04.180162 7905 raft_consensus.cc:729] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 398206deeb46492e87d93151bcf0b252, State: Initialized, Role: FOLLOWER
I20260806 09:26:04.181054 7905 consensus_queue.cc:260] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } }
I20260806 09:26:04.182893 7705 heartbeater.cc:503] Master 127.16.86.190:34327 was elected leader, sending a full tablet report...
I20260806 09:26:04.183684 7905 ts_tablet_manager.cc:1428] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252: Time spent starting tablet: real 0.017s user 0.019s sys 0.000s
I20260806 09:26:04.184002 7908 raft_consensus.cc:348] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } }
I20260806 09:26:04.184535 7908 raft_consensus.cc:374] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:04.184660 7908 raft_consensus.cc:729] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 40e0f22a71424e07bed66412e6d24f19, State: Initialized, Role: FOLLOWER
I20260806 09:26:04.185427 7908 consensus_queue.cc:260] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } }
I20260806 09:26:04.189324 7882 heartbeater.cc:503] Master 127.16.86.190:34327 was elected leader, sending a full tablet report...
I20260806 09:26:04.189163 7907 raft_consensus.cc:348] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } }
I20260806 09:26:04.189785 7907 raft_consensus.cc:374] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:04.189899 7907 raft_consensus.cc:729] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d6da66d904e45219f37ff127f9dc2e4, State: Initialized, Role: FOLLOWER
I20260806 09:26:04.190310 7908 ts_tablet_manager.cc:1428] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19: Time spent starting tablet: real 0.017s user 0.013s sys 0.005s
I20260806 09:26:04.190732 7907 consensus_queue.cc:260] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } }
I20260806 09:26:04.193439 7907 ts_tablet_manager.cc:1428] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4: Time spent starting tablet: real 0.019s user 0.017s sys 0.001s
W20260806 09:26:04.204346 7526 tablet.cc:2369] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:26:04.239642 7883 tablet.cc:2369] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:26:04.324978 7706 tablet.cc:2369] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:26:04.396010 7912 raft_consensus.cc:482] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:26:04.396407 7912 raft_consensus.cc:504] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } }
I20260806 09:26:04.398346 7912 leader_election.cc:288] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 40e0f22a71424e07bed66412e6d24f19 (127.16.86.131:38851), 4d6da66d904e45219f37ff127f9dc2e4 (127.16.86.129:43931)
I20260806 09:26:04.407927 7901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.399235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:52239 (local address 127.16.86.131:38851)
0806 09:26:04.399639 (+ 404us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:04.399647 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:04.400201 (+ 554us) server_negotiation.cc:407] Connection header received
0806 09:26:04.400489 (+ 288us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:04.400496 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:04.400602 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:04.400711 (+ 109us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:04.401633 (+ 922us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.402665 (+ 1032us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.405616 (+ 2951us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.406595 (+ 979us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.406851 (+ 256us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.406972 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:04.407414 (+ 442us) server_negotiation.cc:299] Negotiation successful
0806 09:26:04.407593 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260806 09:26:04.408097 7916 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.398974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:38851 (local address 127.16.86.130:52239)
0806 09:26:04.400004 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:04.400027 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:04.400256 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:04.400914 (+ 658us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:04.400922 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:04.400971 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:04.401463 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:04.401481 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.402832 (+ 1351us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.402838 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:04.405339 (+ 2501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:04.405359 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.406951 (+ 1592us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.406958 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:04.407203 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.407235 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:26:04.407372 (+ 137us) client_negotiation.cc:240] Negotiation successful
0806 09:26:04.407558 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":872,"mutex_wait_us":105,"thread_start_us":547,"threads_started":1}
I20260806 09:26:04.409566 7918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.400230 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:43931 (local address 127.16.86.130:35771)
0806 09:26:04.400991 (+ 761us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:04.401007 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:04.401137 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:04.401604 (+ 467us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:04.401608 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:04.401638 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:04.402142 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:04.402159 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.403847 (+ 1688us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.403851 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:04.406779 (+ 2928us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:04.406797 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.408547 (+ 1750us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.408552 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:04.408706 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.408734 (+ 28us) client_negotiation.cc:769] Sending connection context
0806 09:26:04.408882 (+ 148us) client_negotiation.cc:240] Negotiation successful
0806 09:26:04.409081 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":643,"mutex_wait_us":1,"thread_start_us":387,"threads_started":1}
I20260806 09:26:04.409584 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.400597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:35771 (local address 127.16.86.129:43931)
0806 09:26:04.400881 (+ 284us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:04.400887 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:04.401112 (+ 225us) server_negotiation.cc:407] Connection header received
0806 09:26:04.401288 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:04.401293 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:04.401397 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:04.401537 (+ 140us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:04.402382 (+ 845us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.403659 (+ 1277us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.407121 (+ 3462us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.408022 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.408271 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.408354 (+ 83us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:04.409163 (+ 809us) server_negotiation.cc:299] Negotiation successful
0806 09:26:04.409337 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":113}
I20260806 09:26:04.409444 7824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff2d7646f1448f68894bdde5691eee5" candidate_uuid: "398206deeb46492e87d93151bcf0b252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e0f22a71424e07bed66412e6d24f19" is_pre_election: true
I20260806 09:26:04.410131 7824 raft_consensus.cc:2454] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398206deeb46492e87d93151bcf0b252 in term 0.
I20260806 09:26:04.411262 7553 leader_election.cc:302] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398206deeb46492e87d93151bcf0b252, 40e0f22a71424e07bed66412e6d24f19; no voters:
I20260806 09:26:04.411279 7472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff2d7646f1448f68894bdde5691eee5" candidate_uuid: "398206deeb46492e87d93151bcf0b252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d6da66d904e45219f37ff127f9dc2e4" is_pre_election: true
I20260806 09:26:04.411885 7912 raft_consensus.cc:2781] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:26:04.412096 7912 raft_consensus.cc:482] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:26:04.412050 7472 raft_consensus.cc:2454] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 398206deeb46492e87d93151bcf0b252 in term 0.
I20260806 09:26:04.412253 7912 raft_consensus.cc:3037] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:04.415424 7912 raft_consensus.cc:504] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } }
I20260806 09:26:04.416214 7912 leader_election.cc:288] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [CANDIDATE]: Term 1 election: Requested vote from peers 40e0f22a71424e07bed66412e6d24f19 (127.16.86.131:38851), 4d6da66d904e45219f37ff127f9dc2e4 (127.16.86.129:43931)
I20260806 09:26:04.417076 7824 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff2d7646f1448f68894bdde5691eee5" candidate_uuid: "398206deeb46492e87d93151bcf0b252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "40e0f22a71424e07bed66412e6d24f19"
I20260806 09:26:04.417098 7472 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "fff2d7646f1448f68894bdde5691eee5" candidate_uuid: "398206deeb46492e87d93151bcf0b252" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d6da66d904e45219f37ff127f9dc2e4"
I20260806 09:26:04.417347 7824 raft_consensus.cc:3037] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:04.417394 7472 raft_consensus.cc:3037] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:04.420739 7824 raft_consensus.cc:2454] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398206deeb46492e87d93151bcf0b252 in term 1.
I20260806 09:26:04.420760 7472 raft_consensus.cc:2454] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 398206deeb46492e87d93151bcf0b252 in term 1.
I20260806 09:26:04.421396 7553 leader_election.cc:302] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 398206deeb46492e87d93151bcf0b252, 40e0f22a71424e07bed66412e6d24f19; no voters:
I20260806 09:26:04.421873 7912 raft_consensus.cc:2781] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:26:04.422551 7912 raft_consensus.cc:686] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 1 LEADER]: Becoming Leader. State: Replica: 398206deeb46492e87d93151bcf0b252, State: Running, Role: LEADER
I20260806 09:26:04.423276 7912 consensus_queue.cc:237] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } } peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } }
I20260806 09:26:04.429724 7286 catalog_manager.cc:5495] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 reported cstate change: term changed from 0 to 1, leader changed from <none> to 398206deeb46492e87d93151bcf0b252 (127.16.86.130). New cstate: current_term: 1 leader_uuid: "398206deeb46492e87d93151bcf0b252" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "398206deeb46492e87d93151bcf0b252" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41903 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 } health_report { overall_health: UNKNOWN } } }
I20260806 09:26:04.515803 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.508522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46062 (local address 127.16.86.130:41903)
0806 09:26:04.508759 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:04.508764 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:04.508786 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:26:04.508969 (+ 183us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:04.508973 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:04.509083 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:04.509244 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:26:04.509808 (+ 564us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.510736 (+ 928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.512811 (+ 2075us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.513274 (+ 463us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.513393 (+ 119us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.514160 (+ 767us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:26:04.514862 (+ 702us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:26:04.515071 (+ 209us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:04.515400 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 09:26:04.515502 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:26:04.529075 7472 raft_consensus.cc:1261] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4 [term 1 FOLLOWER]: Refusing update from remote peer 398206deeb46492e87d93151bcf0b252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:26:04.529603 7824 raft_consensus.cc:1261] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19 [term 1 FOLLOWER]: Refusing update from remote peer 398206deeb46492e87d93151bcf0b252: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:26:04.530179 7912 consensus_queue.cc:1035] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d6da66d904e45219f37ff127f9dc2e4" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 43931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:26:04.530705 7919 consensus_queue.cc:1035] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [LEADER]: Connected to new peer: Peer: permanent_uuid: "40e0f22a71424e07bed66412e6d24f19" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 38851 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:26:04.567824 7286 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:26:04.568111 7286 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49026:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:26:04.572986 7286 catalog_manager.cc:5782] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Sending DeleteTablet for 3 replicas of tablet fff2d7646f1448f68894bdde5691eee5
I20260806 09:26:04.574941 7797 tablet_service.cc:1514] Processing DeleteTablet for tablet fff2d7646f1448f68894bdde5691eee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:26:04 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:49550
I20260806 09:26:04.575237 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 7331
I20260806 09:26:04.575304 7616 tablet_service.cc:1514] Processing DeleteTablet for tablet fff2d7646f1448f68894bdde5691eee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:26:04 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:46058
I20260806 09:26:04.576200 7444 tablet_service.cc:1514] Processing DeleteTablet for tablet fff2d7646f1448f68894bdde5691eee5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:26:04 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:49544
I20260806 09:26:04.577459 7937 tablet_replica.cc:330] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4: stopping tablet replica
I20260806 09:26:04.578083 7937 raft_consensus.cc:2229] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:26:04.578748 7937 raft_consensus.cc:2258] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:26:04.581816 7937 ts_tablet_manager.cc:1905] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:26:04.587590 7934 tablet_replica.cc:330] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252: stopping tablet replica
I20260806 09:26:04.588620 7934 raft_consensus.cc:2229] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:26:04.589815 7934 raft_consensus.cc:2258] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:26:04.591042 7933 tablet_replica.cc:330] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19: stopping tablet replica
I20260806 09:26:04.593779 7934 ts_tablet_manager.cc:1905] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:26:04.596401 7937 ts_tablet_manager.cc:1918] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:26:04.596624 7937 log.cc:1196] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/fff2d7646f1448f68894bdde5691eee5
I20260806 09:26:04.597209 7933 raft_consensus.cc:2229] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:26:04.597348 7937 ts_tablet_manager.cc:1939] T fff2d7646f1448f68894bdde5691eee5 P 4d6da66d904e45219f37ff127f9dc2e4: Deleting consensus metadata
I20260806 09:26:04.598299 7933 raft_consensus.cc:2258] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:26:04.600178 7270 catalog_manager.cc:4841] TS 4d6da66d904e45219f37ff127f9dc2e4 (127.16.86.129:43931): tablet fff2d7646f1448f68894bdde5691eee5 (table test-table [id=28d1c821cc8c43b3b51480875861e171]) successfully deleted
I20260806 09:26:04.602281 7933 ts_tablet_manager.cc:1905] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:26:04.608366 7934 ts_tablet_manager.cc:1918] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:26:04.608763 7934 log.cc:1196] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/fff2d7646f1448f68894bdde5691eee5
I20260806 09:26:04.609790 7934 ts_tablet_manager.cc:1939] T fff2d7646f1448f68894bdde5691eee5 P 398206deeb46492e87d93151bcf0b252: Deleting consensus metadata
I20260806 09:26:04.610217 7904 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.576579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49552 (local address 127.16.86.129:43931)
0806 09:26:04.576826 (+ 247us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:04.576833 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:04.586705 (+ 9872us) server_negotiation.cc:407] Connection header received
0806 09:26:04.587128 (+ 423us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:04.587135 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:04.587244 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:04.587348 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:04.589976 (+ 2628us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.591234 (+ 1258us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.593642 (+ 2408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.594397 (+ 755us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.594534 (+ 137us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.599946 (+ 5412us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:04.599964 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:04.599985 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:04.600041 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:04.605141 (+ 5100us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:04.606492 (+ 1351us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:04.606502 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:04.606506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:04.606717 (+ 211us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:04.607346 (+ 629us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:04.607354 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:04.607358 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:04.608356 (+ 998us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:04.608834 (+ 478us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:04.609273 (+ 439us) server_negotiation.cc:299] Negotiation successful
0806 09:26:04.609586 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 09:26:04.611645 7519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:04.613265 7273 catalog_manager.cc:4841] TS 398206deeb46492e87d93151bcf0b252 (127.16.86.130:41903): tablet fff2d7646f1448f68894bdde5691eee5 (table test-table [id=28d1c821cc8c43b3b51480875861e171]) successfully deleted
I20260806 09:26:04.617538 7933 ts_tablet_manager.cc:1918] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:26:04.617803 7933 log.cc:1196] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/fff2d7646f1448f68894bdde5691eee5
I20260806 09:26:04.618589 7933 ts_tablet_manager.cc:1939] T fff2d7646f1448f68894bdde5691eee5 P 40e0f22a71424e07bed66412e6d24f19: Deleting consensus metadata
I20260806 09:26:04.621583 7270 catalog_manager.cc:4841] TS 40e0f22a71424e07bed66412e6d24f19 (127.16.86.131:38851): tablet fff2d7646f1448f68894bdde5691eee5 (table test-table [id=28d1c821cc8c43b3b51480875861e171]) successfully deleted
I20260806 09:26:04.778008 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 7331
I20260806 09:26:04.812956 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 7530
I20260806 09:26:04.830103 7896 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:04.814111 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46074 (local address 127.16.86.130:41903)
0806 09:26:04.814597 (+ 486us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:04.814602 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:04.814618 (+ 16us) server_negotiation.cc:407] Connection header received
0806 09:26:04.814685 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:04.814690 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:04.814758 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:04.814915 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:04.816542 (+ 1627us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.817477 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.819109 (+ 1632us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:04.819593 (+ 484us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:04.819760 (+ 167us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:04.822943 (+ 3183us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:04.822962 (+ 19us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:04.822987 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:04.823087 (+ 100us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:04.826761 (+ 3674us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:04.827627 (+ 866us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:04.827637 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:04.827639 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:04.827798 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:04.828235 (+ 437us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:04.828239 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:04.828241 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:04.828684 (+ 443us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:04.829081 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:04.829405 (+ 324us) server_negotiation.cc:299] Negotiation successful
0806 09:26:04.829672 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360}
I20260806 09:26:04.831037 7699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:04.973750 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 7530
I20260806 09:26:05.012755 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 7710
I20260806 09:26:05.036551 7941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:05.013786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49558 (local address 127.16.86.131:38851)
0806 09:26:05.014434 (+ 648us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:05.014440 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:05.014468 (+ 28us) server_negotiation.cc:407] Connection header received
0806 09:26:05.014540 (+ 72us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:05.014545 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:05.014630 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:05.014767 (+ 137us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:05.017049 (+ 2282us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:05.018564 (+ 1515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:05.020856 (+ 2292us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:05.021635 (+ 779us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:05.021820 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:05.025975 (+ 4155us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:05.025993 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:05.026013 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:05.026058 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:05.030865 (+ 4807us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:05.032855 (+ 1990us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:05.032877 (+ 22us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:05.032880 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:05.033265 (+ 385us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:05.034097 (+ 832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:05.034101 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:05.034103 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:05.034534 (+ 431us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:05.035088 (+ 554us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:05.035838 (+ 750us) server_negotiation.cc:299] Negotiation successful
0806 09:26:05.036062 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":482,"thread_start_us":360,"threads_started":1}
I20260806 09:26:05.037784 7875 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:05.147987 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 7710
I20260806 09:26:05.179332 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 7248
I20260806 09:26:05.180878 7322 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:05.311175 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 7248
I20260806 09:26:25.352408 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:34327
--webserver_interface=127.16.86.190
--webserver_port=39097
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:34327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:26:25.743678 7944 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:25.744036 7944 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:25.744205 7944 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:25.751955 7944 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20260806 09:26:25.752092 7944 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20260806 09:26:25.752192 7944 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20260806 09:26:25.752291 7944 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:26:25.752388 7944 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:25.752512 7944 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:25.752701 7944 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:26:25.752902 7944 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:26:25.765201 7944 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:34327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:34327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=39097
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:25.766680 7944 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:25.768656 7944 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:25.784817 7950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:25.784960 7952 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:25.785919 7949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:25.786242 7944 server_base.cc:1034] running on GCE node
I20260806 09:26:25.787225 7944 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:25.788787 7944 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:25.790059 7944 hybrid_clock.cc:648] HybridClock initialized: now 1786008385789985 us; error 53 us; skew 500 ppm
I20260806 09:26:25.790422 7944 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:25 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008385, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:25.798774 7944 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:26:25.800861 7944 webserver.cc:460] Webserver started at http://127.16.86.190:39097/ using document root <none> and password file <none>
I20260806 09:26:25.801697 7944 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:25.801851 7944 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:25.808187 7944 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:26:25.811827 7963 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:25.813501 7944 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260806 09:26:25.813736 7944 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "e063cec8fc994c39af3bc0a6b3d75ffa"
format_stamp: "Formatted at 2026-08-06 09:25:38 on dist-test-slave-h3ft"
server_key: "a2cfafb68656cf53cdc4ac7d524d1c9f"
server_key_iv: "c464e578db4287f2116d86c96412b70a"
server_key_version: "encryptionkey@0"
I20260806 09:26:25.814786 7944 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:25.831565 7944 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:25.836169 7944 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:25.836494 7944 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:25.859427 7944 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:34327
I20260806 09:26:25.859519 8040 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:34327 every 8 connection(s)
I20260806 09:26:25.861116 7944 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:26:25.863515 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 7944
I20260806 09:26:25.864439 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:43931
--local_ip_for_outbound_sockets=127.16.86.129
--tserver_master_addrs=127.16.86.190:34327
--webserver_port=39767
--webserver_interface=127.16.86.129
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:26:25.870414 8042 sys_catalog.cc:263] Verifying existing consensus state
I20260806 09:26:25.874680 8042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Bootstrap starting.
I20260806 09:26:25.901001 8042 log.cc:824] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:25.920836 8042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260806 09:26:25.921577 8042 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Bootstrap complete.
I20260806 09:26:25.932291 8042 raft_consensus.cc:348] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } }
I20260806 09:26:25.933482 8042 raft_consensus.cc:729] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e063cec8fc994c39af3bc0a6b3d75ffa, State: Initialized, Role: FOLLOWER
I20260806 09:26:25.934290 8042 consensus_queue.cc:260] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } }
I20260806 09:26:25.934616 8042 raft_consensus.cc:388] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:26:25.934775 8042 raft_consensus.cc:482] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:26:25.934984 8042 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 2 FOLLOWER]: Advancing to term 3
I20260806 09:26:25.939970 8042 raft_consensus.cc:504] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } }
I20260806 09:26:25.940717 8042 leader_election.cc:302] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e063cec8fc994c39af3bc0a6b3d75ffa; no voters:
I20260806 09:26:25.941582 8042 leader_election.cc:288] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [CANDIDATE]: Term 3 election: Requested vote from peers
I20260806 09:26:25.941758 8048 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 3 FOLLOWER]: Leader election won for term 3
I20260806 09:26:25.943643 8048 raft_consensus.cc:686] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [term 3 LEADER]: Becoming Leader. State: Replica: e063cec8fc994c39af3bc0a6b3d75ffa, State: Running, Role: LEADER
I20260806 09:26:25.944416 8048 consensus_queue.cc:237] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } }
I20260806 09:26:25.944739 8042 sys_catalog.cc:564] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:26:25.947665 8050 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: SysCatalogTable state changed. Reason: New leader e063cec8fc994c39af3bc0a6b3d75ffa. Latest consensus state: current_term: 3 leader_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } } }
I20260806 09:26:25.948230 8050 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: This master's current role is: LEADER
I20260806 09:26:25.958434 8049 sys_catalog.cc:455] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e063cec8fc994c39af3bc0a6b3d75ffa" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34327 } } }
I20260806 09:26:25.958880 8049 sys_catalog.cc:458] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa [sys.catalog]: This master's current role is: LEADER
I20260806 09:26:25.959081 8054 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:26:25.972280 8054 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:26:25.974153 8054 catalog_manager.cc:1259] Loaded cluster ID: 6e87a7edfa5a4e62865e50dd6680ff96
I20260806 09:26:25.974303 8054 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:26:25.980180 8054 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:26:25.982281 8054 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Loaded TSK: 0
I20260806 09:26:25.982455 8054 catalog_manager.cc:5879] T 00000000000000000000000000000000 P e063cec8fc994c39af3bc0a6b3d75ffa: Loaded TSK: 1
F20260806 09:26:25.983901 8054 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=RSA error:04093004:rsa routines:OLD_RSA_PRIV_DECODE:RSA lib:rsa_ameth.c:121 error:0D0680A8:asn1 encoding routines:ASN1_CHECK_TLEN:wrong tag:tasn_dec.c:1220 error:0D06C03A:asn1 encoding routines:ASN1_D2I_EX_PRIMITIVE:nested asn1 error:tasn_dec.c:788 error:0D08303A:asn1 encoding routines:ASN1_TEMPLATE_NOEXP_D2I:nested asn1 error:tasn_dec.c:720:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fa891a289ad google::LogMessage::Fail() at ??:0
@ 0x7fa891a2b88a google::LogMessage::SendToLog() at ??:0
@ 0x7fa891a284de google::LogMessage::Flush() at ??:0
@ 0x7fa891a29a19 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fa89578c7d2 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fa89577b73e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fa8a74611c0 kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fa8a7460783 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fa8a74ec2a7 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fa8a74ec0c4 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fa8a75365e9 std::function<>::operator()() at ??:0
@ 0x7fa8a7464cdd kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fa8a7462c36 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fa8a74ea610 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fa8a74ea439 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa894674345 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fa89467791c kudu::ThreadPool::CreateThread()::$_2::operator()() at ??:0
@ 0x7fa894677799 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x9edb7d std::function<>::operator()() at ??:?
@ 0x7fa894652eaf kudu::Thread::SuperviseThread() at ??:0
@ 0x7fa89792c6db start_thread at ??:0
@ 0x7fa8908c261f clone at ??:0
W20260806 09:26:26.361645 8044 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:26.362048 8044 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:26.362237 8044 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:26.371654 8044 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:26.371827 8044 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:26.372088 8044 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:26:26.384011 8044 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:43931
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=39767
--tserver_master_addrs=127.16.86.190:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:26.385854 8044 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:26.387639 8044 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:26.405224 8105 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:26.405499 8107 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:26.406117 8109 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:26.406368 8044 server_base.cc:1034] running on GCE node
I20260806 09:26:26.407305 8044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:26.408551 8044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:26.409840 8044 hybrid_clock.cc:648] HybridClock initialized: now 1786008386409748 us; error 85 us; skew 500 ppm
I20260806 09:26:26.410284 8044 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:26 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008386, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:26.414268 8044 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:26:26.416082 8044 webserver.cc:460] Webserver started at http://127.16.86.129:39767/ using document root <none> and password file <none>
I20260806 09:26:26.416878 8044 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:26.416999 8044 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:26.423300 8044 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:26:26.426702 8124 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:26.428824 8044 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:26:26.429014 8044 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "4d6da66d904e45219f37ff127f9dc2e4"
format_stamp: "Formatted at 2026-08-06 09:25:39 on dist-test-slave-h3ft"
server_key: "3bb55387eae8f12ce29f3bfb0897e06b"
server_key_iv: "02b7e1de7c82950ee2880d78645e3666"
server_key_version: "encryptionkey@0"
I20260806 09:26:26.430280 8044 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:26.458317 8044 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:26.463735 8044 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:26.464017 8044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:26.465240 8044 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:26.466856 8044 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:26.466974 8044 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:26.467152 8044 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:26.467231 8044 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:26.511518 8044 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:43931
I20260806 09:26:26.511740 8273 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:43931 every 8 connection(s)
I20260806 09:26:26.513453 8044 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:26:26.520463 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:26.517566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:43143)
0806 09:26:26.518660 (+ 1094us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:26.519212 (+ 552us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":663,"thread_start_us":403,"threads_started":1}
W20260806 09:26:26.521654 8274 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:34327 (0 consecutive failures): Network error: Failed to ping master at 127.16.86.190:34327: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
I20260806 09:26:26.521987 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 8044
I20260806 09:26:26.522981 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:26.522400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:45015)
0806 09:26:26.522649 (+ 249us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:26.522778 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 09:26:26.523005 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:41903
--local_ip_for_outbound_sockets=127.16.86.130
--tserver_master_addrs=127.16.86.190:34327
--webserver_port=45413
--webserver_interface=127.16.86.130
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:26:26.524386 8276 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:26.523824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:34493)
0806 09:26:26.524048 (+ 224us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:26.524177 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
W20260806 09:26:26.524781 8274 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:26:26.920382 8278 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:26.920763 8278 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:26.920955 8278 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:26.929162 8278 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:26.929337 8278 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:26.929541 8278 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:26:26.941006 8278 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:41903
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=45413
--tserver_master_addrs=127.16.86.190:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:26.942525 8278 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:26.944192 8278 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:26.961313 8289 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:26.961256 8287 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:26.961385 8286 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:26.962266 8278 server_base.cc:1034] running on GCE node
I20260806 09:26:26.962996 8278 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:26.964231 8278 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:26.965515 8278 hybrid_clock.cc:648] HybridClock initialized: now 1786008386965473 us; error 37 us; skew 500 ppm
I20260806 09:26:26.965955 8278 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:26 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008386, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:26.969779 8278 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:26:26.971630 8278 webserver.cc:460] Webserver started at http://127.16.86.130:45413/ using document root <none> and password file <none>
I20260806 09:26:26.972386 8278 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:26.972556 8278 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:26.978405 8278 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:26:26.981711 8296 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:26.983633 8278 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:26:26.983887 8278 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "398206deeb46492e87d93151bcf0b252"
format_stamp: "Formatted at 2026-08-06 09:25:39 on dist-test-slave-h3ft"
server_key: "826cb9293b0c4a2ed89f6ac7614c4239"
server_key_iv: "7442e7b4e151955c9327a4e2ff1fda8f"
server_key_version: "encryptionkey@0"
I20260806 09:26:26.985069 8278 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:27.001645 8278 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:27.006608 8278 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:27.006929 8278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:27.008147 8278 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:27.009701 8278 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:27.009843 8278 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:27.009976 8278 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:27.010078 8278 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:27.050882 8278 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:41903
I20260806 09:26:27.050984 8441 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:41903 every 8 connection(s)
I20260806 09:26:27.052618 8278 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:26:27.056216 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 8278
I20260806 09:26:27.057044 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:38851
--local_ip_for_outbound_sockets=127.16.86.131
--tserver_master_addrs=127.16.86.190:34327
--webserver_port=38961
--webserver_interface=127.16.86.131
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:38213
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:26:27.059194 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:27.056759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:42345)
0806 09:26:27.057754 (+ 995us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:27.058103 (+ 349us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":665,"thread_start_us":439,"threads_started":1}
W20260806 09:26:27.060379 8442 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:34327 (0 consecutive failures): Network error: Failed to ping master at 127.16.86.190:34327: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
I20260806 09:26:27.061725 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:27.061223 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:51741)
0806 09:26:27.061456 (+ 233us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:27.061569 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20260806 09:26:27.062836 8446 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:27.062444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:59821)
0806 09:26:27.062607 (+ 163us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:27.062711 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 09:26:27.063176 8442 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20260806 09:26:27.487567 8447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:27.487984 8447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:27.488211 8447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:27.496722 8447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:27.496886 8447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:27.497085 8447 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:26:27.508376 8447 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:38213
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:38851
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=38961
--tserver_master_addrs=127.16.86.190:34327
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:27.510152 8447 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:27.511937 8447 file_cache.cc:493] Constructed file cache file cache with capacity 419430
I20260806 09:26:27.526456 8459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:27.525500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:35469)
0806 09:26:27.526094 (+ 594us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:27.526199 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":316,"threads_started":1}
W20260806 09:26:27.528337 8455 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:27.528409 8456 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:27.528581 8458 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:27.530961 8447 server_base.cc:1034] running on GCE node
I20260806 09:26:27.531847 8447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:27.533092 8447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:27.534318 8447 hybrid_clock.cc:648] HybridClock initialized: now 1786008387534219 us; error 79 us; skew 500 ppm
I20260806 09:26:27.534775 8447 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:27 dist-test-slave-h3ft krb5kdc[6525](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008387, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:27.538978 8447 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:26:27.540868 8447 webserver.cc:460] Webserver started at http://127.16.86.131:38961/ using document root <none> and password file <none>
I20260806 09:26:27.541754 8447 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:27.541917 8447 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:27.548945 8447 fs_manager.cc:709] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:26:27.552834 8466 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:27.555135 8447 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260806 09:26:27.555373 8447 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "40e0f22a71424e07bed66412e6d24f19"
format_stamp: "Formatted at 2026-08-06 09:25:40 on dist-test-slave-h3ft"
server_key: "4306d4d44746e5f12dafdfa827c0536f"
server_key_iv: "88307b3f26b4221c0a35802d1a42c23c"
server_key_version: "encryptionkey@0"
I20260806 09:26:27.556612 8447 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:27.575150 8447 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:27.580291 8447 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:27.580595 8447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:27.581774 8447 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:27.583385 8447 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:27.583515 8447 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:27.583659 8447 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:27.583752 8447 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:27.625877 8447 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:38851
I20260806 09:26:27.625934 8620 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:38851 every 8 connection(s)
I20260806 09:26:27.628139 8447 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:26:27.631314 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 8447
I20260806 09:26:27.635131 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:27.631980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:55715)
0806 09:26:27.633251 (+ 1271us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:27.633709 (+ 458us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":742,"thread_start_us":474,"threads_started":1}
W20260806 09:26:27.636030 8622 heartbeater.cc:644] Failed to heartbeat to 127.16.86.190:34327 (0 consecutive failures): Network error: Failed to ping master at 127.16.86.190:34327: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
I20260806 09:26:27.637018 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:27.636621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:54281)
0806 09:26:27.636797 (+ 176us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:27.636894 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
I20260806 09:26:27.638170 8625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:27.637756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:35355)
0806 09:26:27.637946 (+ 190us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:27.638050 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
W20260806 09:26:27.638478 8622 heartbeater.cc:415] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20260806 09:26:28.065442 8627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:28.064048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:34613)
0806 09:26:28.064917 (+ 869us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:28.065104 (+ 187us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":668,"thread_start_us":436,"threads_started":1}
I20260806 09:26:28.528662 8628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:28.527561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:38647)
0806 09:26:28.528284 (+ 723us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:28.528422 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":568,"thread_start_us":350,"threads_started":1}
I20260806 09:26:28.640287 8631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:28.639271 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:42997)
0806 09:26:28.639910 (+ 639us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:28.640041 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":282,"threads_started":1}
I20260806 09:26:29.068003 8633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:29.066897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:38551)
0806 09:26:29.067596 (+ 699us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:29.067720 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":358,"threads_started":1}
I20260806 09:26:29.530789 8634 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:29.529828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:46663)
0806 09:26:29.530427 (+ 599us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:29.530535 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":325,"threads_started":1}
I20260806 09:26:29.642450 8635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:29.641543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:50139)
0806 09:26:29.642157 (+ 614us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:29.642258 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":309,"threads_started":1}
I20260806 09:26:30.070374 8636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:30.069339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:45977)
0806 09:26:30.070034 (+ 695us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:30.070155 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":549,"thread_start_us":393,"threads_started":1}
I20260806 09:26:30.532974 8637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:30.532018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:35161)
0806 09:26:30.532673 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:30.532782 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":508,"thread_start_us":358,"threads_started":1}
I20260806 09:26:30.644799 8639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:30.643725 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:48491)
0806 09:26:30.644464 (+ 739us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:30.644589 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":604,"thread_start_us":367,"threads_started":1}
I20260806 09:26:31.072808 8640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:31.071764 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:60257)
0806 09:26:31.072477 (+ 713us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:31.072594 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":505,"thread_start_us":330,"threads_started":1}
I20260806 09:26:31.535279 8641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:31.534236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:49791)
0806 09:26:31.534859 (+ 623us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:31.535065 (+ 206us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":328,"threads_started":1}
I20260806 09:26:31.647153 8642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:31.646011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:37315)
0806 09:26:31.646733 (+ 722us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:31.646852 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":370,"threads_started":1}
I20260806 09:26:32.075323 8643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:32.074168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:51955)
0806 09:26:32.074948 (+ 780us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:32.075120 (+ 172us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":628,"thread_start_us":358,"threads_started":1}
I20260806 09:26:32.537287 8644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:32.536325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:53457)
0806 09:26:32.536938 (+ 613us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:32.537073 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":317,"threads_started":1}
I20260806 09:26:32.649389 8645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:32.648358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:57313)
0806 09:26:32.649045 (+ 687us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:32.649164 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":544,"thread_start_us":335,"threads_started":1}
I20260806 09:26:33.077574 8646 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:33.076636 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:41111)
0806 09:26:33.077265 (+ 629us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:33.077388 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":314,"threads_started":1}
I20260806 09:26:33.539732 8647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:33.538525 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:46913)
0806 09:26:33.539315 (+ 790us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:33.539442 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":639,"thread_start_us":331,"threads_started":1}
I20260806 09:26:33.651809 8648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:33.650727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:39731)
0806 09:26:33.651474 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:33.651598 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":368,"threads_started":1}
I20260806 09:26:34.080533 8650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:34.079318 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:48217)
0806 09:26:34.080154 (+ 836us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:34.080276 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":655,"thread_start_us":345,"threads_started":1}
I20260806 09:26:34.541927 8652 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:34.540997 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:33129)
0806 09:26:34.541598 (+ 601us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:34.541728 (+ 130us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":305,"threads_started":1}
I20260806 09:26:34.654376 8653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:34.653188 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:38149)
0806 09:26:34.653954 (+ 766us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:34.654089 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":594,"thread_start_us":342,"threads_started":1}
I20260806 09:26:35.083252 8654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:35.082016 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:35015)
0806 09:26:35.082762 (+ 746us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:35.082928 (+ 166us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":385,"threads_started":1}
I20260806 09:26:35.544250 8655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:35.543343 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:53489)
0806 09:26:35.543955 (+ 612us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:35.544053 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":463,"thread_start_us":285,"threads_started":1}
I20260806 09:26:35.657055 8656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:35.655981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:44811)
0806 09:26:35.656657 (+ 676us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:35.656783 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":353,"threads_started":1}
I20260806 09:26:36.085587 8657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:36.084567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:59083)
0806 09:26:36.085264 (+ 697us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:36.085373 (+ 109us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":354,"threads_started":1}
I20260806 09:26:36.546492 8658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:36.545481 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:37875)
0806 09:26:36.546149 (+ 668us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:36.546272 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":522,"thread_start_us":355,"threads_started":1}
I20260806 09:26:36.659510 8659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:36.658345 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:50797)
0806 09:26:36.659171 (+ 826us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:36.659296 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":596,"thread_start_us":382,"threads_started":1}
I20260806 09:26:37.088126 8661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:37.087185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:35133)
0806 09:26:37.087828 (+ 643us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:37.087943 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":341,"threads_started":1}
I20260806 09:26:37.548878 8662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:37.547857 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:52009)
0806 09:26:37.548530 (+ 673us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:37.548646 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":547,"thread_start_us":352,"threads_started":1}
I20260806 09:26:37.662254 8663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:37.661039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:53247)
0806 09:26:37.661891 (+ 852us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:37.662031 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":685,"thread_start_us":484,"threads_started":1}
I20260806 09:26:38.090634 8664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:38.089571 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:53403)
0806 09:26:38.090290 (+ 719us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:38.090421 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":335,"threads_started":1}
I20260806 09:26:38.551215 8665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:38.550197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:37461)
0806 09:26:38.550830 (+ 633us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:38.550974 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":506,"thread_start_us":295,"threads_started":1}
I20260806 09:26:38.664880 8667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:38.663738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:54713)
0806 09:26:38.664497 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:38.664637 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":395,"threads_started":1}
I20260806 09:26:39.093206 8668 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:39.092068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:46663)
0806 09:26:39.092901 (+ 833us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:39.093019 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":319,"threads_started":1}
I20260806 09:26:39.553561 8671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:39.552459 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:57673)
0806 09:26:39.553190 (+ 731us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:39.553304 (+ 114us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":598,"thread_start_us":414,"threads_started":1}
I20260806 09:26:39.667577 8672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:39.666293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:37223)
0806 09:26:39.667132 (+ 839us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:39.667270 (+ 138us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":504,"thread_start_us":343,"threads_started":1}
I20260806 09:26:40.095870 8673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:40.094676 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:46527)
0806 09:26:40.095435 (+ 759us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:40.095592 (+ 157us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":606,"thread_start_us":408,"threads_started":1}
I20260806 09:26:40.555569 8674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:40.554541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:57401)
0806 09:26:40.555196 (+ 655us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:40.555323 (+ 127us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":509,"thread_start_us":274,"threads_started":1}
I20260806 09:26:40.669827 8676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:40.668723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:52435)
0806 09:26:40.669438 (+ 715us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:40.669570 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":557,"thread_start_us":352,"threads_started":1}
I20260806 09:26:41.098456 8679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:41.097294 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:47253)
0806 09:26:41.098014 (+ 720us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:41.098170 (+ 156us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":554,"thread_start_us":358,"threads_started":1}
I20260806 09:26:41.557816 8680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:41.556937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:58539)
0806 09:26:41.557494 (+ 557us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:41.557613 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":419,"thread_start_us":306,"threads_started":1}
I20260806 09:26:41.672117 8681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:41.671138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:34427)
0806 09:26:41.671795 (+ 657us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:41.671903 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":311,"threads_started":1}
I20260806 09:26:42.101297 8682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:42.100289 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.130:43677)
0806 09:26:42.100958 (+ 669us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:42.101075 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":534,"thread_start_us":354,"threads_started":1}
I20260806 09:26:42.560149 8684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:42.558982 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.129:42047)
0806 09:26:42.559763 (+ 781us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:42.559906 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":613,"thread_start_us":290,"threads_started":1}
I20260806 09:26:42.637387 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 8044
I20260806 09:26:42.667375 8685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:42.639233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58754 (local address 127.16.86.129:43931)
0806 09:26:42.640228 (+ 995us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:42.640256 (+ 28us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:42.640338 (+ 82us) server_negotiation.cc:407] Connection header received
0806 09:26:42.642551 (+ 2213us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:42.642564 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:42.643250 (+ 686us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:42.643657 (+ 407us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:42.645565 (+ 1908us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:42.646919 (+ 1354us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:42.648594 (+ 1675us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:42.649435 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:42.649619 (+ 184us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:42.658295 (+ 8676us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:42.658308 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:42.658334 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:42.658406 (+ 72us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:42.662473 (+ 4067us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:42.663617 (+ 1144us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:42.663627 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:42.663638 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:42.663899 (+ 261us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:42.664419 (+ 520us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:42.664423 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:42.664425 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:42.665210 (+ 785us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:42.665523 (+ 313us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:42.666223 (+ 700us) server_negotiation.cc:299] Negotiation successful
0806 09:26:42.666855 (+ 632us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":722,"thread_start_us":439,"threads_started":1}
I20260806 09:26:42.669402 8269 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:42.674738 8687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:42.673666 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:34327 (local address 127.16.86.131:35437)
0806 09:26:42.674283 (+ 617us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:42.674452 (+ 169us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.190:34327: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":327,"threads_started":1}
I20260806 09:26:42.768736 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 8044
I20260806 09:26:42.800320 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 8278
I20260806 09:26:42.822643 8688 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:42.801701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36086 (local address 127.16.86.130:41903)
0806 09:26:42.802357 (+ 656us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:42.802377 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:42.802429 (+ 52us) server_negotiation.cc:407] Connection header received
0806 09:26:42.804330 (+ 1901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:42.804341 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:42.804899 (+ 558us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:42.805287 (+ 388us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:42.806984 (+ 1697us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:42.808528 (+ 1544us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:42.810207 (+ 1679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:42.811208 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:42.811406 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:42.814590 (+ 3184us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:42.814600 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:42.814628 (+ 28us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:42.814692 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:42.818422 (+ 3730us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:42.819344 (+ 922us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:42.819353 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:42.819362 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:42.819612 (+ 250us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:42.819990 (+ 378us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:42.819994 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:42.819996 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:42.820754 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:42.821287 (+ 533us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:42.821667 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 09:26:42.822271 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":304,"threads_started":1}
I20260806 09:26:42.824574 8437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:42.924333 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 8278
I20260806 09:26:42.952381 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 8447
I20260806 09:26:42.974403 8690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:42.953812 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42186 (local address 127.16.86.131:38851)
0806 09:26:42.954579 (+ 767us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:42.954599 (+ 20us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:42.954658 (+ 59us) server_negotiation.cc:407] Connection header received
0806 09:26:42.957059 (+ 2401us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:42.957071 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:42.957703 (+ 632us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:42.958105 (+ 402us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:42.959620 (+ 1515us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:42.961397 (+ 1777us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:42.962916 (+ 1519us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:42.963794 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:42.963952 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:42.966902 (+ 2950us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:42.966910 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:42.966933 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:42.966993 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:42.970725 (+ 3732us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:42.971607 (+ 882us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:42.971616 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:42.971624 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:42.971819 (+ 195us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:42.972208 (+ 389us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:42.972211 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:42.972213 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:42.972882 (+ 669us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:42.973164 (+ 282us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:42.973594 (+ 430us) server_negotiation.cc:299] Negotiation successful
0806 09:26:42.974032 (+ 438us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":553,"thread_start_us":353,"threads_started":1}
I20260806 09:26:42.976310 8616 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:43.066866 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 8447
2026-08-06T09:26:43Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (66982 ms)
[----------] 30 tests from SecurityITest (341496 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:26:43 dist-test-slave-h3ft krb5kdc[8695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:26:43 dist-test-slave-h3ft krb5kdc[8695](info): set up 2 sockets
Aug 06 09:26:43 dist-test-slave-h3ft krb5kdc[8695](info): commencing operation
krb5kdc: starting...
W20260806 09:26:44.466846 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.340s user 0.002s sys 0.050s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:26:44 dist-test-slave-h3ft krb5kdc[8695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:26:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:26:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:26:44.846333 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:36527
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:36527
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:26:45.277613 8714 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:45.278010 8714 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:45.278189 8714 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:45.286420 8714 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:26:45.286576 8714 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:45.286629 8714 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:45.286738 8714 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:26:45.286829 8714 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:26:45.298532 8714 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:36527
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:36527
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:45.300419 8714 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:45.302145 8714 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:45.317265 8719 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:45.317366 8720 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:45.317369 8722 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:45.318912 8714 server_base.cc:1034] running on GCE node
I20260806 09:26:45.319998 8714 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:45.321931 8714 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:45.323226 8714 hybrid_clock.cc:648] HybridClock initialized: now 1786008405323119 us; error 78 us; skew 500 ppm
I20260806 09:26:45.323657 8714 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:45 dist-test-slave-h3ft krb5kdc[8695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008405, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:45.327773 8714 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:26:45.329751 8714 webserver.cc:460] Webserver started at http://127.16.86.190:43759/ using document root <none> and password file <none>
I20260806 09:26:45.330588 8714 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:45.330751 8714 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:45.331163 8714 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:45.334527 8714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "a5bd934bd123456c9384f498bb2fd8c9"
format_stamp: "Formatted at 2026-08-06 09:26:45 on dist-test-slave-h3ft"
server_key: "56b1827cc92bf3e1a37ad01f0928867d"
server_key_iv: "f425d13f037c8367539662832a94de18"
server_key_version: "encryptionkey@0"
I20260806 09:26:45.335413 8714 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "a5bd934bd123456c9384f498bb2fd8c9"
format_stamp: "Formatted at 2026-08-06 09:26:45 on dist-test-slave-h3ft"
server_key: "56b1827cc92bf3e1a37ad01f0928867d"
server_key_iv: "f425d13f037c8367539662832a94de18"
server_key_version: "encryptionkey@0"
I20260806 09:26:45.341295 8714 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260806 09:26:45.345273 8733 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:45.346853 8714 fs_manager.cc:725] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260806 09:26:45.347095 8714 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "a5bd934bd123456c9384f498bb2fd8c9"
format_stamp: "Formatted at 2026-08-06 09:26:45 on dist-test-slave-h3ft"
server_key: "56b1827cc92bf3e1a37ad01f0928867d"
server_key_iv: "f425d13f037c8367539662832a94de18"
server_key_version: "encryptionkey@0"
I20260806 09:26:45.347330 8714 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:45.362874 8714 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:45.367802 8714 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:45.368162 8714 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:45.391286 8714 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:36527
I20260806 09:26:45.391340 8808 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:36527 every 8 connection(s)
I20260806 09:26:45.392946 8714 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:26:45.397938 8810 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:45.399150 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 8714
I20260806 09:26:45.399444 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:26:45.400455 16730 external_mini_cluster.cc:1428] Setting key 7c9ba856e301d9cb8950fa352302ac57
I20260806 09:26:45.416774 8810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9: Bootstrap starting.
Aug 06 09:26:45 dist-test-slave-h3ft krb5kdc[8695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:26:45.421257 8810 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:45.422623 8810 log.cc:824] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:45.426494 8810 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9: No bootstrap required, opened a new log
I20260806 09:26:45.429386 8813 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:45.403270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47952 (local address 127.16.86.190:36527)
0806 09:26:45.404308 (+ 1038us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:45.404322 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:45.404375 (+ 53us) server_negotiation.cc:407] Connection header received
0806 09:26:45.406352 (+ 1977us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:45.406382 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:45.406866 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:45.407336 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:45.409663 (+ 2327us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:45.411137 (+ 1474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:45.413410 (+ 2273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:45.414283 (+ 873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:45.414507 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:45.420426 (+ 5919us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:45.420435 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:45.420454 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:45.420512 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:45.424615 (+ 4103us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:45.425495 (+ 880us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:45.425503 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:45.425510 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:45.425715 (+ 205us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:45.426119 (+ 404us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:45.426126 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:45.426129 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:45.426828 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:45.427222 (+ 394us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:45.427720 (+ 498us) server_negotiation.cc:299] Negotiation successful
0806 09:26:45.428194 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":783,"thread_start_us":516,"threads_started":1}
I20260806 09:26:45.436477 8810 raft_consensus.cc:348] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5bd934bd123456c9384f498bb2fd8c9" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36527 } }
I20260806 09:26:45.436988 8810 raft_consensus.cc:374] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:45.437111 8810 raft_consensus.cc:729] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a5bd934bd123456c9384f498bb2fd8c9, State: Initialized, Role: FOLLOWER
I20260806 09:26:45.437863 8810 consensus_queue.cc:260] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5bd934bd123456c9384f498bb2fd8c9" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36527 } }
I20260806 09:26:45.438160 8810 raft_consensus.cc:388] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:26:45.438324 8810 raft_consensus.cc:482] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:26:45.438526 8810 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:45.441524 8810 raft_consensus.cc:504] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5bd934bd123456c9384f498bb2fd8c9" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36527 } }
I20260806 09:26:45.442207 8810 leader_election.cc:302] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a5bd934bd123456c9384f498bb2fd8c9; no voters:
I20260806 09:26:45.443054 8810 leader_election.cc:288] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:26:45.443279 8815 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:26:45.444813 8815 raft_consensus.cc:686] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [term 1 LEADER]: Becoming Leader. State: Replica: a5bd934bd123456c9384f498bb2fd8c9, State: Running, Role: LEADER
I20260806 09:26:45.445505 8815 consensus_queue.cc:237] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5bd934bd123456c9384f498bb2fd8c9" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36527 } }
I20260806 09:26:45.445914 8810 sys_catalog.cc:564] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:26:45.449019 8816 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a5bd934bd123456c9384f498bb2fd8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5bd934bd123456c9384f498bb2fd8c9" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36527 } } }
I20260806 09:26:45.449540 8816 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [sys.catalog]: This master's current role is: LEADER
I20260806 09:26:45.449852 8817 sys_catalog.cc:455] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a5bd934bd123456c9384f498bb2fd8c9. Latest consensus state: current_term: 1 leader_uuid: "a5bd934bd123456c9384f498bb2fd8c9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a5bd934bd123456c9384f498bb2fd8c9" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36527 } } }
I20260806 09:26:45.450197 8817 sys_catalog.cc:458] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9 [sys.catalog]: This master's current role is: LEADER
I20260806 09:26:45.454255 8827 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:26:45.460903 8827 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:26:45.471698 8827 catalog_manager.cc:1347] Generated new cluster ID: b3928753c482497daf730804aace029d
I20260806 09:26:45.471860 8827 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:26:45.495100 8827 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:26:45.496753 8827 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:26:45.511751 8827 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9: Generated new TSK 0
I20260806 09:26:45.512944 8827 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:26:45.613032 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36527
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:26:46.060926 8842 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:46.061306 8842 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:46.061416 8842 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:46.069816 8842 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:46.069990 8842 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:46.070128 8842 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:26:46.081677 8842 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:46.083469 8842 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:46.085078 8842 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:46.102218 8853 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:46.102293 8850 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:46.102329 8849 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:46.105080 8842 server_base.cc:1034] running on GCE node
I20260806 09:26:46.106011 8842 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:46.107421 8842 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:46.108709 8842 hybrid_clock.cc:648] HybridClock initialized: now 1786008406108652 us; error 50 us; skew 500 ppm
I20260806 09:26:46.109138 8842 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:46 dist-test-slave-h3ft krb5kdc[8695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008406, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:46.113384 8842 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:26:46.115633 8842 webserver.cc:460] Webserver started at http://127.16.86.129:33311/ using document root <none> and password file <none>
I20260806 09:26:46.116698 8842 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:46.116871 8842 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:46.117278 8842 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:46.120998 8842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696"
format_stamp: "Formatted at 2026-08-06 09:26:46 on dist-test-slave-h3ft"
server_key: "f4c4d2d3e578c6d11dcb72ca132a2da2"
server_key_iv: "99e74db4e437a6bf6fe884d5fdc37352"
server_key_version: "encryptionkey@0"
I20260806 09:26:46.121870 8842 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696"
format_stamp: "Formatted at 2026-08-06 09:26:46 on dist-test-slave-h3ft"
server_key: "f4c4d2d3e578c6d11dcb72ca132a2da2"
server_key_iv: "99e74db4e437a6bf6fe884d5fdc37352"
server_key_version: "encryptionkey@0"
I20260806 09:26:46.128368 8842 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:26:46.132921 8865 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:46.134775 8842 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260806 09:26:46.135044 8842 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696"
format_stamp: "Formatted at 2026-08-06 09:26:46 on dist-test-slave-h3ft"
server_key: "f4c4d2d3e578c6d11dcb72ca132a2da2"
server_key_iv: "99e74db4e437a6bf6fe884d5fdc37352"
server_key_version: "encryptionkey@0"
I20260806 09:26:46.135356 8842 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:46.152076 8842 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:46.157040 8842 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:46.157413 8842 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:46.158737 8842 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:46.160528 8842 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:46.160614 8842 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:46.160768 8842 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:46.160851 8842 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:46.203235 8842 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:37373
I20260806 09:26:46.203356 9015 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:37373 every 8 connection(s)
I20260806 09:26:46.204952 8842 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:26:46.210773 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 8842
I20260806 09:26:46.211086 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:26:46.211977 16730 external_mini_cluster.cc:1428] Setting key deeef8f9cf52ecfb37e158e039000788
Aug 06 09:26:46 dist-test-slave-h3ft krb5kdc[8695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008406, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:26:46.231921 9018 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:46.208297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:57799 (local address 127.16.86.190:36527)
0806 09:26:46.208816 (+ 519us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:46.208820 (+ 4us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:46.209896 (+ 1076us) server_negotiation.cc:407] Connection header received
0806 09:26:46.212980 (+ 3084us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:46.212985 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:46.213089 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:46.213244 (+ 155us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:46.215766 (+ 2522us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:46.216825 (+ 1059us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:46.218766 (+ 1941us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:46.219359 (+ 593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:46.219535 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:46.224114 (+ 4579us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:46.224120 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:46.224124 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:46.224169 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:46.227413 (+ 3244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:46.228299 (+ 886us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:46.228307 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:46.228314 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:46.228493 (+ 179us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:46.229305 (+ 812us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:46.229312 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:46.229317 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:46.229775 (+ 458us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:46.229980 (+ 205us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:46.230985 (+ 1005us) server_negotiation.cc:299] Negotiation successful
0806 09:26:46.231323 (+ 338us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":293,"threads_started":1}
I20260806 09:26:46.232728 9019 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:46.208789 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:36527 (local address 127.16.86.129:57799)
0806 09:26:46.209691 (+ 902us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:46.209739 (+ 48us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:46.212610 (+ 2871us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:46.213635 (+ 1025us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:46.213646 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:46.214539 (+ 893us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:46.215464 (+ 925us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:46.215488 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:46.217027 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:46.217033 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:46.218571 (+ 1538us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:46.218587 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:46.219655 (+ 1068us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:46.219666 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:46.219987 (+ 321us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:46.220969 (+ 982us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:26:46.220997 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:46.223775 (+ 2778us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:46.227601 (+ 3826us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:46.227613 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:46.227636 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:46.228145 (+ 509us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:46.228748 (+ 603us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:46.228755 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:46.228758 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:46.229115 (+ 357us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:46.229993 (+ 878us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:46.230001 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:46.230469 (+ 468us) client_negotiation.cc:769] Sending connection context
0806 09:26:46.230791 (+ 322us) client_negotiation.cc:240] Negotiation successful
0806 09:26:46.231419 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":595,"thread_start_us":333,"threads_started":1}
I20260806 09:26:46.234764 9016 heartbeater.cc:347] Connected to a master server at 127.16.86.190:36527
I20260806 09:26:46.235190 9016 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:46.235997 9016 heartbeater.cc:511] Master 127.16.86.190:36527 requested a full tablet report, sending...
I20260806 09:26:46.238356 8755 ts_manager.cc:194] Registered new tserver with Master: ab8c6a42a9ce4f7e9a3c45f3096c0696 (127.16.86.129:37373)
I20260806 09:26:46.240202 8755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:57799
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:26:46.299983 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36527
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:26:46.701928 9027 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:46.702255 9027 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:46.702438 9027 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:46.710107 9027 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:46.710249 9027 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:46.710460 9027 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:26:46.720924 9027 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:46.722394 9027 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:46.723917 9027 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:46.739756 9038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:46.739931 9036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:46.740253 9033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:46.741539 9027 server_base.cc:1034] running on GCE node
I20260806 09:26:46.742244 9027 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:46.743414 9027 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:46.744692 9027 hybrid_clock.cc:648] HybridClock initialized: now 1786008406744636 us; error 52 us; skew 500 ppm
I20260806 09:26:46.745102 9027 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:46 dist-test-slave-h3ft krb5kdc[8695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008406, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:46.749028 9027 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:26:46.750972 9027 webserver.cc:460] Webserver started at http://127.16.86.130:37947/ using document root <none> and password file <none>
I20260806 09:26:46.751924 9027 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:46.752096 9027 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:46.752470 9027 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:46.755776 9027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "3929a953e7bb450fa43e425b851ce076"
format_stamp: "Formatted at 2026-08-06 09:26:46 on dist-test-slave-h3ft"
server_key: "bf10360be679c17efa4119e51b46b48a"
server_key_iv: "273709b4bbee3db3096685262cb4a702"
server_key_version: "encryptionkey@0"
I20260806 09:26:46.756671 9027 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "3929a953e7bb450fa43e425b851ce076"
format_stamp: "Formatted at 2026-08-06 09:26:46 on dist-test-slave-h3ft"
server_key: "bf10360be679c17efa4119e51b46b48a"
server_key_iv: "273709b4bbee3db3096685262cb4a702"
server_key_version: "encryptionkey@0"
I20260806 09:26:46.762812 9027 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260806 09:26:46.766983 9047 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:46.769040 9027 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:26:46.769217 9027 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "3929a953e7bb450fa43e425b851ce076"
format_stamp: "Formatted at 2026-08-06 09:26:46 on dist-test-slave-h3ft"
server_key: "bf10360be679c17efa4119e51b46b48a"
server_key_iv: "273709b4bbee3db3096685262cb4a702"
server_key_version: "encryptionkey@0"
I20260806 09:26:46.769455 9027 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:46.793671 9027 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:46.798988 9027 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:46.799376 9027 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:46.800648 9027 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:46.802345 9027 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:46.802436 9027 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:46.802542 9027 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:46.802595 9027 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:46.848223 9027 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:46039
I20260806 09:26:46.848390 9192 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:46039 every 8 connection(s)
I20260806 09:26:46.850457 9027 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:26:46.860436 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 9027
I20260806 09:26:46.860703 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:26:46.861235 16730 external_mini_cluster.cc:1428] Setting key 953a1c21cc53eb54d06b33cf316c9ea0
Aug 06 09:26:46 dist-test-slave-h3ft krb5kdc[8695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008406, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:26:46.879097 9195 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:46.855091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:46939 (local address 127.16.86.190:36527)
0806 09:26:46.855767 (+ 676us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:46.855772 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:46.857155 (+ 1383us) server_negotiation.cc:407] Connection header received
0806 09:26:46.859582 (+ 2427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:46.859588 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:46.859699 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:46.859862 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:46.862051 (+ 2189us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:46.863165 (+ 1114us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:46.865298 (+ 2133us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:46.866151 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:46.866356 (+ 205us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:46.870439 (+ 4083us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:46.870449 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:46.870457 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:46.870517 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:46.874695 (+ 4178us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:46.875812 (+ 1117us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:46.875820 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:46.875827 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:46.876037 (+ 210us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:46.876558 (+ 521us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:46.876565 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:46.876571 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:46.877034 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:46.877222 (+ 188us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:46.878171 (+ 949us) server_negotiation.cc:299] Negotiation successful
0806 09:26:46.878496 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":481,"thread_start_us":315,"threads_started":1}
I20260806 09:26:46.879997 9196 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:46.855878 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:36527 (local address 127.16.86.130:46939)
0806 09:26:46.856897 (+ 1019us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:46.856970 (+ 73us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:46.859226 (+ 2256us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:46.860185 (+ 959us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:46.860195 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:46.860991 (+ 796us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:46.861760 (+ 769us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:46.861788 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:46.863377 (+ 1589us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:46.863383 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:46.865079 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:46.865094 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:46.866442 (+ 1348us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:46.866447 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:46.866727 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:46.867633 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:26:46.867659 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:46.870116 (+ 2457us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:46.874918 (+ 4802us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:46.874926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:46.874940 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:46.875592 (+ 652us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:46.876189 (+ 597us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:46.876193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:46.876195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:46.876411 (+ 216us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:46.877203 (+ 792us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:46.877210 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:46.877704 (+ 494us) client_negotiation.cc:769] Sending connection context
0806 09:26:46.877988 (+ 284us) client_negotiation.cc:240] Negotiation successful
0806 09:26:46.878508 (+ 520us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":675,"thread_start_us":385,"threads_started":1}
I20260806 09:26:46.881947 9193 heartbeater.cc:347] Connected to a master server at 127.16.86.190:36527
I20260806 09:26:46.882339 9193 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:46.883293 9193 heartbeater.cc:511] Master 127.16.86.190:36527 requested a full tablet report, sending...
I20260806 09:26:46.885265 8755 ts_manager.cc:194] Registered new tserver with Master: 3929a953e7bb450fa43e425b851ce076 (127.16.86.130:46039)
I20260806 09:26:46.886718 8755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:46939
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
W20260806 09:26:46.967481 16730 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.86.131: real 0.103s user 0.005s sys 0.032s
I20260806 09:26:46.968525 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36527
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:34827
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:26:47.245204 9016 heartbeater.cc:503] Master 127.16.86.190:36527 was elected leader, sending a full tablet report...
W20260806 09:26:47.359194 9200 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:47.359704 9200 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:47.359920 9200 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:47.368538 9200 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:47.368752 9200 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:47.369099 9200 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:26:47.380056 9200 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:34827
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:36527
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:47.381649 9200 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:47.383139 9200 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:47.403855 9213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:47.403929 9211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:47.404088 9209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:47.404913 9200 server_base.cc:1034] running on GCE node
I20260806 09:26:47.405752 9200 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:47.407100 9200 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:47.408362 9200 hybrid_clock.cc:648] HybridClock initialized: now 1786008407408309 us; error 43 us; skew 500 ppm
I20260806 09:26:47.408710 9200 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:47 dist-test-slave-h3ft krb5kdc[8695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008407, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:47.412727 9200 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:26:47.414620 9200 webserver.cc:460] Webserver started at http://127.16.86.131:42613/ using document root <none> and password file <none>
I20260806 09:26:47.415522 9200 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:47.415650 9200 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:47.416074 9200 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:47.419348 9200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "a399abdbe72148eba5b3bb2ec2aaf127"
format_stamp: "Formatted at 2026-08-06 09:26:47 on dist-test-slave-h3ft"
server_key: "a9e4424bddba2f3f5bed30e85440dffe"
server_key_iv: "b2d31e14e6c1df7242c71ddb8bbd6d58"
server_key_version: "encryptionkey@0"
I20260806 09:26:47.420343 9200 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "a399abdbe72148eba5b3bb2ec2aaf127"
format_stamp: "Formatted at 2026-08-06 09:26:47 on dist-test-slave-h3ft"
server_key: "a9e4424bddba2f3f5bed30e85440dffe"
server_key_iv: "b2d31e14e6c1df7242c71ddb8bbd6d58"
server_key_version: "encryptionkey@0"
I20260806 09:26:47.426609 9200 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 09:26:47.431288 9223 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:47.433655 9200 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:26:47.433847 9200 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "a399abdbe72148eba5b3bb2ec2aaf127"
format_stamp: "Formatted at 2026-08-06 09:26:47 on dist-test-slave-h3ft"
server_key: "a9e4424bddba2f3f5bed30e85440dffe"
server_key_iv: "b2d31e14e6c1df7242c71ddb8bbd6d58"
server_key_version: "encryptionkey@0"
I20260806 09:26:47.434119 9200 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:47.461971 9200 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:47.467921 9200 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:47.468327 9200 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:47.469756 9200 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:47.471432 9200 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:47.471524 9200 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:47.471697 9200 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:47.471755 9200 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:47.514439 9200 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:42111
I20260806 09:26:47.514564 9376 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:42111 every 8 connection(s)
I20260806 09:26:47.515985 9200 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:26:47.520295 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 9200
I20260806 09:26:47.520541 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:26:47.521200 16730 external_mini_cluster.cc:1428] Setting key 83ce6861f790051571c71ac27e6af5d4
Aug 06 09:26:47 dist-test-slave-h3ft krb5kdc[8695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008407, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:26:47.541465 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.519614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:36377 (local address 127.16.86.190:36527)
0806 09:26:47.520323 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:47.520330 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:47.521568 (+ 1238us) server_negotiation.cc:407] Connection header received
0806 09:26:47.524111 (+ 2543us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:47.524117 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:47.524224 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:47.524357 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:47.526197 (+ 1840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.527140 (+ 943us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.528743 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.529165 (+ 422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.529345 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.533372 (+ 4027us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:47.533378 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:47.533381 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:47.533417 (+ 36us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:47.537270 (+ 3853us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:47.538253 (+ 983us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:47.538261 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:47.538265 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:47.538533 (+ 268us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:47.539218 (+ 685us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:47.539223 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:47.539227 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:47.539747 (+ 520us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:47.539909 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:47.540802 (+ 893us) server_negotiation.cc:299] Negotiation successful
0806 09:26:47.541114 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":323,"threads_started":1}
I20260806 09:26:47.542388 9381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.520321 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:36527 (local address 127.16.86.131:36377)
0806 09:26:47.521338 (+ 1017us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:47.521399 (+ 61us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:47.523781 (+ 2382us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:47.524626 (+ 845us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:47.524635 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:47.525353 (+ 718us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:47.525983 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.526009 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.527313 (+ 1304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.527317 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.528574 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.528587 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.529439 (+ 852us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.529446 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.529714 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.530456 (+ 742us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:26:47.530479 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:47.533127 (+ 2648us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:47.537472 (+ 4345us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:47.537483 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:47.537508 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:47.538066 (+ 558us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:47.538722 (+ 656us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:47.538727 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:47.538731 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:47.539064 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:47.539896 (+ 832us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:47.539903 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:47.540312 (+ 409us) client_negotiation.cc:769] Sending connection context
0806 09:26:47.540588 (+ 276us) client_negotiation.cc:240] Negotiation successful
0806 09:26:47.541143 (+ 555us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":702,"thread_start_us":451,"threads_started":1}
I20260806 09:26:47.544183 9377 heartbeater.cc:347] Connected to a master server at 127.16.86.190:36527
I20260806 09:26:47.544684 9377 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:47.545699 9377 heartbeater.cc:511] Master 127.16.86.190:36527 requested a full tablet report, sending...
I20260806 09:26:47.547793 8755 ts_manager.cc:194] Registered new tserver with Master: a399abdbe72148eba5b3bb2ec2aaf127 (127.16.86.131:42111)
I20260806 09:26:47.548666 8755 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:36377
I20260806 09:26:47.552536 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:26:47.573784 9379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.557801 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47960 (local address 127.16.86.190:36527)
0806 09:26:47.558185 (+ 384us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:47.558198 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:47.558692 (+ 494us) server_negotiation.cc:407] Connection header received
0806 09:26:47.558903 (+ 211us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:47.558908 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:47.559083 (+ 175us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:47.559201 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:47.560980 (+ 1779us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.561915 (+ 935us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.563571 (+ 1656us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.564032 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.564149 (+ 117us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.566959 (+ 2810us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:47.566983 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:47.566987 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:47.567130 (+ 143us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:47.570492 (+ 3362us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:47.571362 (+ 870us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:47.571367 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:47.571370 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:47.571531 (+ 161us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:47.571955 (+ 424us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:47.571958 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:47.571966 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:47.572356 (+ 390us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:47.572612 (+ 256us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:47.573047 (+ 435us) server_negotiation.cc:299] Negotiation successful
0806 09:26:47.573356 (+ 309us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":150}
I20260806 09:26:47.579577 8755 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:26:47.586894 8755 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:26:47.628388 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.604340 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35070 (local address 127.16.86.131:42111)
0806 09:26:47.618521 (+ 14181us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:47.618530 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:47.618578 (+ 48us) server_negotiation.cc:407] Connection header received
0806 09:26:47.618710 (+ 132us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:47.618719 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:47.619334 (+ 615us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:47.621123 (+ 1789us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:47.621500 (+ 377us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.622732 (+ 1232us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.625767 (+ 3035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.627060 (+ 1293us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.627223 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.627500 (+ 277us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:47.627801 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 09:26:47.627983 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13971,"thread_start_us":13765,"threads_started":1}
I20260806 09:26:47.628508 9396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.603139 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:42111 (local address 127.0.0.1:35070)
0806 09:26:47.616808 (+ 13669us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:47.616827 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:47.616968 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:47.619559 (+ 2591us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:47.619563 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:47.619588 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:47.620885 (+ 1297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.620909 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.622939 (+ 2030us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.622945 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.625556 (+ 2611us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.625574 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.627284 (+ 1710us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.627291 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.627498 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.627566 (+ 68us) client_negotiation.cc:769] Sending connection context
0806 09:26:47.627746 (+ 180us) client_negotiation.cc:240] Negotiation successful
0806 09:26:47.627984 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13546,"mutex_wait_us":638,"thread_start_us":12013,"threads_started":1}
I20260806 09:26:47.629156 9400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.603689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:37373 (local address 127.0.0.1:35150)
0806 09:26:47.619086 (+ 15397us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:47.619117 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:47.619449 (+ 332us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:47.620497 (+ 1048us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:47.620503 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:47.620533 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:47.621129 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.621143 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.623164 (+ 2021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.623169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.625899 (+ 2730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.625913 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.628238 (+ 2325us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.628243 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.628423 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.628457 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:26:47.628557 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 09:26:47.628703 (+ 146us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15192,"thread_start_us":14964,"threads_started":1}
I20260806 09:26:47.629675 9395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.603103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:46039 (local address 127.0.0.1:38248)
0806 09:26:47.615774 (+ 12671us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:47.615831 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:47.616135 (+ 304us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:47.619447 (+ 3312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:47.619478 (+ 31us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:47.619523 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:47.620258 (+ 735us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.620276 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.622479 (+ 2203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.622485 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.625390 (+ 2905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.625407 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.627946 (+ 2539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.627953 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.628196 (+ 243us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.628242 (+ 46us) client_negotiation.cc:769] Sending connection context
0806 09:26:47.628344 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 09:26:47.628948 (+ 604us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12274,"mutex_wait_us":78,"thread_start_us":13958,"threads_started":1}
I20260806 09:26:47.629887 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.603352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38248 (local address 127.16.86.130:46039)
0806 09:26:47.618561 (+ 15209us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:47.618571 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:47.618598 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:26:47.618709 (+ 111us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:47.618719 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:47.619111 (+ 392us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:47.619952 (+ 841us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:47.620436 (+ 484us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.622282 (+ 1846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.625644 (+ 3362us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.627345 (+ 1701us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.627513 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.627762 (+ 249us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:47.629271 (+ 1509us) server_negotiation.cc:299] Negotiation successful
0806 09:26:47.629481 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14995,"thread_start_us":14815,"threads_started":1}
I20260806 09:26:47.630594 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.604450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35150 (local address 127.16.86.129:37373)
0806 09:26:47.617946 (+ 13496us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:47.617961 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:47.619290 (+ 1329us) server_negotiation.cc:407] Connection header received
0806 09:26:47.619678 (+ 388us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:47.619690 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:47.619992 (+ 302us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:47.620352 (+ 360us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:47.621459 (+ 1107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.622953 (+ 1494us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.626106 (+ 3153us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.627664 (+ 1558us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.627821 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.628072 (+ 251us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:47.630077 (+ 2005us) server_negotiation.cc:299] Negotiation successful
0806 09:26:47.630289 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13282,"thread_start_us":13086,"threads_started":1}
I20260806 09:26:47.634133 9284 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7db5b88b5543cba30a3513a1d4b9d3 (DEFAULT_TABLE table=test-table [id=9308c6e7994f4dbba5f36668297a3cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:47.634223 9110 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7db5b88b5543cba30a3513a1d4b9d3 (DEFAULT_TABLE table=test-table [id=9308c6e7994f4dbba5f36668297a3cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:47.634828 8932 tablet_service.cc:1467] Processing CreateTablet for tablet 1a7db5b88b5543cba30a3513a1d4b9d3 (DEFAULT_TABLE table=test-table [id=9308c6e7994f4dbba5f36668297a3cf7]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:47.636024 9284 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7db5b88b5543cba30a3513a1d4b9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:47.636248 9110 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7db5b88b5543cba30a3513a1d4b9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:47.636478 8932 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 1a7db5b88b5543cba30a3513a1d4b9d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:47.651383 9404 tablet_bootstrap.cc:492] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076: Bootstrap starting.
I20260806 09:26:47.656030 9404 tablet_bootstrap.cc:654] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:47.656428 9405 tablet_bootstrap.cc:492] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696: Bootstrap starting.
I20260806 09:26:47.657140 9406 tablet_bootstrap.cc:492] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127: Bootstrap starting.
I20260806 09:26:47.657925 9404 log.cc:824] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:47.661499 9404 tablet_bootstrap.cc:492] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076: No bootstrap required, opened a new log
I20260806 09:26:47.661837 9406 tablet_bootstrap.cc:654] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:47.661885 9404 ts_tablet_manager.cc:1397] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20260806 09:26:47.662951 9405 tablet_bootstrap.cc:654] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:47.663591 9406 log.cc:824] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:47.664510 9405 log.cc:824] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:47.667477 9406 tablet_bootstrap.cc:492] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127: No bootstrap required, opened a new log
I20260806 09:26:47.667878 9406 ts_tablet_manager.cc:1397] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127: Time spent bootstrapping tablet: real 0.011s user 0.006s sys 0.004s
I20260806 09:26:47.670630 9405 tablet_bootstrap.cc:492] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696: No bootstrap required, opened a new log
I20260806 09:26:47.670995 9405 ts_tablet_manager.cc:1397] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696: Time spent bootstrapping tablet: real 0.015s user 0.007s sys 0.000s
I20260806 09:26:47.671586 9404 raft_consensus.cc:348] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3929a953e7bb450fa43e425b851ce076" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46039 } } peers { permanent_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 42111 } } peers { permanent_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 37373 } }
I20260806 09:26:47.672042 9404 raft_consensus.cc:374] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:47.672163 9404 raft_consensus.cc:729] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3929a953e7bb450fa43e425b851ce076, State: Initialized, Role: FOLLOWER
I20260806 09:26:47.672892 9404 consensus_queue.cc:260] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3929a953e7bb450fa43e425b851ce076" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46039 } } peers { permanent_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 42111 } } peers { permanent_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 37373 } }
I20260806 09:26:47.674319 9193 heartbeater.cc:503] Master 127.16.86.190:36527 was elected leader, sending a full tablet report...
I20260806 09:26:47.674922 9404 ts_tablet_manager.cc:1428] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076: Time spent starting tablet: real 0.013s user 0.014s sys 0.000s
I20260806 09:26:47.680028 9406 raft_consensus.cc:348] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3929a953e7bb450fa43e425b851ce076" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46039 } } peers { permanent_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 42111 } } peers { permanent_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 37373 } }
I20260806 09:26:47.680616 9406 raft_consensus.cc:374] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:47.680735 9406 raft_consensus.cc:729] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a399abdbe72148eba5b3bb2ec2aaf127, State: Initialized, Role: FOLLOWER
I20260806 09:26:47.681687 9406 consensus_queue.cc:260] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3929a953e7bb450fa43e425b851ce076" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46039 } } peers { permanent_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 42111 } } peers { permanent_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 37373 } }
I20260806 09:26:47.683187 9377 heartbeater.cc:503] Master 127.16.86.190:36527 was elected leader, sending a full tablet report...
I20260806 09:26:47.684918 9405 raft_consensus.cc:348] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3929a953e7bb450fa43e425b851ce076" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46039 } } peers { permanent_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 42111 } } peers { permanent_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 37373 } }
I20260806 09:26:47.685451 9405 raft_consensus.cc:374] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:47.685590 9405 raft_consensus.cc:729] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab8c6a42a9ce4f7e9a3c45f3096c0696, State: Initialized, Role: FOLLOWER
I20260806 09:26:47.685860 9406 ts_tablet_manager.cc:1428] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127: Time spent starting tablet: real 0.018s user 0.015s sys 0.000s
I20260806 09:26:47.686527 9405 consensus_queue.cc:260] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3929a953e7bb450fa43e425b851ce076" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46039 } } peers { permanent_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 42111 } } peers { permanent_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 37373 } }
I20260806 09:26:47.688591 9405 ts_tablet_manager.cc:1428] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260806 09:26:47.708781 9017 tablet.cc:2369] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:26:47.727990 9417 raft_consensus.cc:482] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:26:47.728395 9417 raft_consensus.cc:504] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3929a953e7bb450fa43e425b851ce076" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46039 } } peers { permanent_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 42111 } } peers { permanent_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 37373 } }
I20260806 09:26:47.730592 9417 leader_election.cc:288] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3929a953e7bb450fa43e425b851ce076 (127.16.86.130:46039), a399abdbe72148eba5b3bb2ec2aaf127 (127.16.86.131:42111)
I20260806 09:26:47.731657 9413 raft_consensus.cc:482] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:26:47.732585 9413 raft_consensus.cc:504] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3929a953e7bb450fa43e425b851ce076" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46039 } } peers { permanent_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 42111 } } peers { permanent_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 37373 } }
I20260806 09:26:47.736742 9413 leader_election.cc:288] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers a399abdbe72148eba5b3bb2ec2aaf127 (127.16.86.131:42111), ab8c6a42a9ce4f7e9a3c45f3096c0696 (127.16.86.129:37373)
I20260806 09:26:47.744081 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.731909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:35995 (local address 127.16.86.131:42111)
0806 09:26:47.732260 (+ 351us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:47.732271 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:47.733477 (+ 1206us) server_negotiation.cc:407] Connection header received
0806 09:26:47.733629 (+ 152us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:47.733635 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:47.733737 (+ 102us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:47.733882 (+ 145us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:47.734644 (+ 762us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.735928 (+ 1284us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.740464 (+ 4536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.742392 (+ 1928us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.742595 (+ 203us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.742727 (+ 132us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:47.743555 (+ 828us) server_negotiation.cc:299] Negotiation successful
0806 09:26:47.743726 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":104}
I20260806 09:26:47.744701 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.731326 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:37349 (local address 127.16.86.130:46039)
0806 09:26:47.731650 (+ 324us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:47.731659 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:47.732908 (+ 1249us) server_negotiation.cc:407] Connection header received
0806 09:26:47.733336 (+ 428us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:47.733345 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:47.733456 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:47.733532 (+ 76us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:47.734622 (+ 1090us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.736037 (+ 1415us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.740753 (+ 4716us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.742609 (+ 1856us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.742795 (+ 186us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.742922 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:47.744166 (+ 1244us) server_negotiation.cc:299] Negotiation successful
0806 09:26:47.744339 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":117}
I20260806 09:26:47.745226 9421 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.731322 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:42111 (local address 127.16.86.129:35995)
0806 09:26:47.733315 (+ 1993us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:47.733340 (+ 25us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:47.733480 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:47.733924 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:47.733929 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:47.733970 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:47.734443 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.734456 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.736203 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.736213 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.740112 (+ 3899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.740147 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.743055 (+ 2908us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.743066 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.743329 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.743365 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:26:47.744523 (+ 1158us) client_negotiation.cc:240] Negotiation successful
0806 09:26:47.744701 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1850,"mutex_wait_us":2,"thread_start_us":1567,"threads_started":1}
I20260806 09:26:47.746356 9422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.731224 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:46039 (local address 127.16.86.129:37349)
0806 09:26:47.732704 (+ 1480us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:47.732730 (+ 26us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:47.733104 (+ 374us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:47.733756 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:47.733759 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:47.733781 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:47.734377 (+ 596us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.734395 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.736345 (+ 1950us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.736356 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.740338 (+ 3982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.740375 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.743617 (+ 3242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.743626 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.743897 (+ 271us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.743959 (+ 62us) client_negotiation.cc:769] Sending connection context
0806 09:26:47.744929 (+ 970us) client_negotiation.cc:240] Negotiation successful
0806 09:26:47.745180 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1280,"mutex_wait_us":637,"thread_start_us":1007,"threads_started":1}
I20260806 09:26:47.746855 9318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7db5b88b5543cba30a3513a1d4b9d3" candidate_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" is_pre_election: true
I20260806 09:26:47.747637 9318 raft_consensus.cc:2454] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab8c6a42a9ce4f7e9a3c45f3096c0696 in term 0.
I20260806 09:26:47.747563 9135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7db5b88b5543cba30a3513a1d4b9d3" candidate_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3929a953e7bb450fa43e425b851ce076" is_pre_election: true
I20260806 09:26:47.748437 8867 leader_election.cc:302] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a399abdbe72148eba5b3bb2ec2aaf127, ab8c6a42a9ce4f7e9a3c45f3096c0696; no voters:
I20260806 09:26:47.748529 9135 raft_consensus.cc:2454] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ab8c6a42a9ce4f7e9a3c45f3096c0696 in term 0.
I20260806 09:26:47.748884 9417 raft_consensus.cc:2781] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:26:47.749042 9417 raft_consensus.cc:482] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:26:47.749145 9417 raft_consensus.cc:3037] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:47.752856 9417 raft_consensus.cc:504] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3929a953e7bb450fa43e425b851ce076" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46039 } } peers { permanent_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 42111 } } peers { permanent_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 37373 } }
I20260806 09:26:47.754102 9417 leader_election.cc:288] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [CANDIDATE]: Term 1 election: Requested vote from peers 3929a953e7bb450fa43e425b851ce076 (127.16.86.130:46039), a399abdbe72148eba5b3bb2ec2aaf127 (127.16.86.131:42111)
I20260806 09:26:47.754755 9318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7db5b88b5543cba30a3513a1d4b9d3" candidate_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a399abdbe72148eba5b3bb2ec2aaf127"
I20260806 09:26:47.754803 9135 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7db5b88b5543cba30a3513a1d4b9d3" candidate_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3929a953e7bb450fa43e425b851ce076"
I20260806 09:26:47.755043 9318 raft_consensus.cc:3037] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:47.755160 9135 raft_consensus.cc:3037] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:47.756633 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.737729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:36251 (local address 127.16.86.129:37373)
0806 09:26:47.738035 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:47.738052 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:47.747569 (+ 9517us) server_negotiation.cc:407] Connection header received
0806 09:26:47.747858 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:47.747864 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:47.747951 (+ 87us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:47.748020 (+ 69us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:47.749021 (+ 1001us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.750321 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.753671 (+ 3350us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.755062 (+ 1391us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.755223 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.755297 (+ 74us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:47.756166 (+ 869us) server_negotiation.cc:299] Negotiation successful
0806 09:26:47.756316 (+ 150us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":123}
I20260806 09:26:47.756680 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.735674 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:43191 (local address 127.16.86.131:42111)
0806 09:26:47.744402 (+ 8728us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:47.744408 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:47.746764 (+ 2356us) server_negotiation.cc:407] Connection header received
0806 09:26:47.747230 (+ 466us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:47.747237 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:47.747329 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:47.747399 (+ 70us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:47.749586 (+ 2187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.750669 (+ 1083us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.753467 (+ 2798us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.754924 (+ 1457us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.755207 (+ 283us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.755297 (+ 90us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:47.756118 (+ 821us) server_negotiation.cc:299] Negotiation successful
0806 09:26:47.756293 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8614,"thread_start_us":10071,"threads_started":1}
I20260806 09:26:47.757663 9423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.735353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:42111 (local address 127.16.86.130:43191)
0806 09:26:47.746615 (+ 11262us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:47.746645 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:47.746987 (+ 342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:47.747658 (+ 671us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:47.747663 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:47.747688 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:47.749408 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.749423 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.750897 (+ 1474us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.750938 (+ 41us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.753286 (+ 2348us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.753301 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.755606 (+ 2305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.755615 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.755864 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.755905 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:26:47.756948 (+ 1043us) client_negotiation.cc:240] Negotiation successful
0806 09:26:47.757080 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11061,"mutex_wait_us":3,"thread_start_us":10772,"threads_started":1}
I20260806 09:26:47.757699 9425 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.737027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:37373 (local address 127.16.86.130:36251)
0806 09:26:47.747451 (+ 10424us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:47.747482 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:47.747666 (+ 184us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:47.748232 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:47.748236 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:47.748262 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:47.748807 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.748821 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.750503 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.750510 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.753285 (+ 2775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:47.753307 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.755750 (+ 2443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.755757 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:47.755946 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.756012 (+ 66us) client_negotiation.cc:769] Sending connection context
0806 09:26:47.756868 (+ 856us) client_negotiation.cc:240] Negotiation successful
0806 09:26:47.757061 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10279,"thread_start_us":10745,"threads_started":1}
I20260806 09:26:47.758752 9318 raft_consensus.cc:2454] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab8c6a42a9ce4f7e9a3c45f3096c0696 in term 1.
I20260806 09:26:47.759384 8867 leader_election.cc:302] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a399abdbe72148eba5b3bb2ec2aaf127, ab8c6a42a9ce4f7e9a3c45f3096c0696; no voters:
I20260806 09:26:47.759476 9135 raft_consensus.cc:2454] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ab8c6a42a9ce4f7e9a3c45f3096c0696 in term 1.
I20260806 09:26:47.759299 8963 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7db5b88b5543cba30a3513a1d4b9d3" candidate_uuid: "3929a953e7bb450fa43e425b851ce076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" is_pre_election: true
I20260806 09:26:47.759991 8963 raft_consensus.cc:2379] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3929a953e7bb450fa43e425b851ce076 in current term 1: Already voted for candidate ab8c6a42a9ce4f7e9a3c45f3096c0696 in this term.
I20260806 09:26:47.760085 9318 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "1a7db5b88b5543cba30a3513a1d4b9d3" candidate_uuid: "3929a953e7bb450fa43e425b851ce076" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" is_pre_election: true
I20260806 09:26:47.760315 9417 raft_consensus.cc:2781] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:26:47.760479 9318 raft_consensus.cc:2379] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 3929a953e7bb450fa43e425b851ce076 in current term 1: Already voted for candidate ab8c6a42a9ce4f7e9a3c45f3096c0696 in this term.
I20260806 09:26:47.761018 9417 raft_consensus.cc:686] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 1 LEADER]: Becoming Leader. State: Replica: ab8c6a42a9ce4f7e9a3c45f3096c0696, State: Running, Role: LEADER
I20260806 09:26:47.761410 9050 leader_election.cc:302] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 3929a953e7bb450fa43e425b851ce076; no voters: a399abdbe72148eba5b3bb2ec2aaf127, ab8c6a42a9ce4f7e9a3c45f3096c0696
I20260806 09:26:47.761708 9417 consensus_queue.cc:237] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3929a953e7bb450fa43e425b851ce076" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46039 } } peers { permanent_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 42111 } } peers { permanent_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 37373 } }
I20260806 09:26:47.761886 9413 raft_consensus.cc:2726] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260806 09:26:47.767983 9378 tablet.cc:2369] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:26:47.767632 8755 catalog_manager.cc:5495] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 reported cstate change: term changed from 0 to 1, leader changed from <none> to ab8c6a42a9ce4f7e9a3c45f3096c0696 (127.16.86.129). New cstate: current_term: 1 leader_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3929a953e7bb450fa43e425b851ce076" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46039 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 42111 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "ab8c6a42a9ce4f7e9a3c45f3096c0696" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 37373 } health_report { overall_health: HEALTHY } } }
I20260806 09:26:47.789793 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.782984 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35166 (local address 127.16.86.129:37373)
0806 09:26:47.783257 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:47.783265 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:47.783399 (+ 134us) server_negotiation.cc:407] Connection header received
0806 09:26:47.783763 (+ 364us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:47.783768 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:47.783871 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:47.784074 (+ 203us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:26:47.784567 (+ 493us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.785456 (+ 889us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.787069 (+ 1613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.787551 (+ 482us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.787710 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.788368 (+ 658us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:26:47.789053 (+ 685us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:26:47.789238 (+ 185us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:47.789306 (+ 68us) server_negotiation.cc:299] Negotiation successful
0806 09:26:47.789408 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":124}
I20260806 09:26:47.800411 9318 raft_consensus.cc:1261] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127 [term 1 FOLLOWER]: Refusing update from remote peer ab8c6a42a9ce4f7e9a3c45f3096c0696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:26:47.800438 9135 raft_consensus.cc:1261] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [term 1 FOLLOWER]: Refusing update from remote peer ab8c6a42a9ce4f7e9a3c45f3096c0696: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:26:47.801549 9426 consensus_queue.cc:1035] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3929a953e7bb450fa43e425b851ce076" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 46039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:26:47.801965 9417 consensus_queue.cc:1035] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a399abdbe72148eba5b3bb2ec2aaf127" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 42111 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:26:47.833279 8754 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:26:47.833650 8754 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:47960:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:26:47.838632 8754 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a5bd934bd123456c9384f498bb2fd8c9: Sending DeleteTablet for 3 replicas of tablet 1a7db5b88b5543cba30a3513a1d4b9d3
I20260806 09:26:47.840123 9284 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7db5b88b5543cba30a3513a1d4b9d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:26:47 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:35070
I20260806 09:26:47.840365 9110 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7db5b88b5543cba30a3513a1d4b9d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:26:47 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:38248
I20260806 09:26:47.840492 8932 tablet_service.cc:1514] Processing DeleteTablet for tablet 1a7db5b88b5543cba30a3513a1d4b9d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:26:47 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:35150
I20260806 09:26:47.851761 9446 tablet_replica.cc:330] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696: stopping tablet replica
I20260806 09:26:47.853492 9446 raft_consensus.cc:2229] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:26:47.854014 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 8842
I20260806 09:26:47.863400 9446 raft_consensus.cc:2258] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:26:47.864650 9443 tablet_replica.cc:330] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127: stopping tablet replica
I20260806 09:26:47.865257 9443 raft_consensus.cc:2229] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:26:47.865933 9443 raft_consensus.cc:2258] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:26:47.866943 9444 tablet_replica.cc:330] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076: stopping tablet replica
I20260806 09:26:47.867417 9444 raft_consensus.cc:2229] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:26:47.867957 9444 raft_consensus.cc:2258] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:26:47.868923 9443 ts_tablet_manager.cc:1905] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:26:47.874325 9446 ts_tablet_manager.cc:1905] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:26:47.877344 9444 ts_tablet_manager.cc:1905] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:26:47.884565 9446 ts_tablet_manager.cc:1918] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:26:47.884938 9446 log.cc:1196] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/1a7db5b88b5543cba30a3513a1d4b9d3
I20260806 09:26:47.885309 9444 ts_tablet_manager.cc:1918] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:26:47.885540 9444 log.cc:1196] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/1a7db5b88b5543cba30a3513a1d4b9d3
I20260806 09:26:47.885897 9446 ts_tablet_manager.cc:1939] T 1a7db5b88b5543cba30a3513a1d4b9d3 P ab8c6a42a9ce4f7e9a3c45f3096c0696: Deleting consensus metadata
I20260806 09:26:47.886252 9444 ts_tablet_manager.cc:1939] T 1a7db5b88b5543cba30a3513a1d4b9d3 P 3929a953e7bb450fa43e425b851ce076: Deleting consensus metadata
I20260806 09:26:47.888742 9443 ts_tablet_manager.cc:1918] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:26:47.888919 8738 catalog_manager.cc:4841] TS ab8c6a42a9ce4f7e9a3c45f3096c0696 (127.16.86.129:37373): tablet 1a7db5b88b5543cba30a3513a1d4b9d3 (table test-table [id=9308c6e7994f4dbba5f36668297a3cf7]) successfully deleted
I20260806 09:26:47.889123 9443 log.cc:1196] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/1a7db5b88b5543cba30a3513a1d4b9d3
I20260806 09:26:47.889680 9443 ts_tablet_manager.cc:1939] T 1a7db5b88b5543cba30a3513a1d4b9d3 P a399abdbe72148eba5b3bb2ec2aaf127: Deleting consensus metadata
Aug 06 09:26:47 dist-test-slave-h3ft krb5kdc[8695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:26:47.891808 8735 catalog_manager.cc:4841] TS a399abdbe72148eba5b3bb2ec2aaf127 (127.16.86.131:42111): tablet 1a7db5b88b5543cba30a3513a1d4b9d3 (table test-table [id=9308c6e7994f4dbba5f36668297a3cf7]) successfully deleted
I20260806 09:26:47.893596 8736 catalog_manager.cc:4841] TS 3929a953e7bb450fa43e425b851ce076 (127.16.86.130:46039): tablet 1a7db5b88b5543cba30a3513a1d4b9d3 (table test-table [id=9308c6e7994f4dbba5f36668297a3cf7]) successfully deleted
I20260806 09:26:47.902318 9403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:47.856813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35170 (local address 127.16.86.129:37373)
0806 09:26:47.857103 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:47.857110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:47.868652 (+ 11542us) server_negotiation.cc:407] Connection header received
0806 09:26:47.869052 (+ 400us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:47.869059 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:47.869176 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:47.869262 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:47.871659 (+ 2397us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.872994 (+ 1335us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.875354 (+ 2360us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:47.876189 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:47.876339 (+ 150us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:47.891992 (+ 15653us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:47.892010 (+ 18us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:47.892035 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:47.892110 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:47.897750 (+ 5640us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:47.899068 (+ 1318us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:47.899079 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:47.899084 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:47.899278 (+ 194us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:47.899839 (+ 561us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:47.899848 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:47.899852 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:47.900694 (+ 842us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:47.901136 (+ 442us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:47.901558 (+ 422us) server_negotiation.cc:299] Negotiation successful
0806 09:26:47.901892 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20260806 09:26:47.903442 9011 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:48.021245 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 8842
I20260806 09:26:48.049922 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 9027
Aug 06 09:26:48 dist-test-slave-h3ft krb5kdc[8695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:26:48.067437 9399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:48.051022 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38258 (local address 127.16.86.130:46039)
0806 09:26:48.051263 (+ 241us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:48.051271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:48.051347 (+ 76us) server_negotiation.cc:407] Connection header received
0806 09:26:48.051645 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:48.051650 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:48.051724 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:48.051856 (+ 132us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:48.053335 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:48.054052 (+ 717us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:48.055531 (+ 1479us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:48.055959 (+ 428us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:48.056121 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:48.061228 (+ 5107us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:48.061244 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:48.061261 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:48.061310 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:48.064376 (+ 3066us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:48.065111 (+ 735us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:48.065119 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:48.065121 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:48.065265 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:48.065695 (+ 430us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:48.065699 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:48.065701 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:48.066123 (+ 422us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:48.066470 (+ 347us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:48.066799 (+ 329us) server_negotiation.cc:299] Negotiation successful
0806 09:26:48.067057 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:26:48.068415 9187 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:48.167030 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 9027
I20260806 09:26:48.199148 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 9200
Aug 06 09:26:48 dist-test-slave-h3ft krb5kdc[8695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008404, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:26:48.217751 9402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:48.200255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35076 (local address 127.16.86.131:42111)
0806 09:26:48.200531 (+ 276us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:48.200538 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:48.200563 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:26:48.200767 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:48.200773 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:48.200874 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:48.201004 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:48.202628 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:48.203567 (+ 939us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:48.204943 (+ 1376us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:48.205417 (+ 474us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:48.205573 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:48.210795 (+ 5222us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:48.210816 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:48.210839 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:48.210884 (+ 45us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:48.214034 (+ 3150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:48.214933 (+ 899us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:48.214942 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:48.214945 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:48.215358 (+ 413us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:48.215835 (+ 477us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:48.215840 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:48.215844 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:48.216344 (+ 500us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:48.216741 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:48.217064 (+ 323us) server_negotiation.cc:299] Negotiation successful
0806 09:26:48.217344 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:26:48.218550 9371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:48.312464 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 9200
I20260806 09:26:48.338470 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 8714
I20260806 09:26:48.339946 8804 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:48.472617 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 8714
2026-08-06T09:26:48Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5401 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2026-08-06T09:26:48Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:26:48Z Disabled control of system clock
I20260806 09:26:48.595580 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:38433
--webserver_interface=127.16.86.190
--webserver_port=0
--builtin_ntp_servers=127.16.86.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:38433
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:26:48.975653 9457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:48.975975 9457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:48.976125 9457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:48.983642 9457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:26:48.983772 9457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:48.983814 9457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:48.983963 9457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:26:48.984035 9457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:26:48.993789 9457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:38433
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:38433
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:48.995281 9457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:48.997654 9457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:49.011723 9464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:49.011765 9465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:49.011823 9467 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:49.013494 9457 server_base.cc:1034] running on GCE node
I20260806 09:26:49.014446 9457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:49.016144 9457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:49.017424 9457 hybrid_clock.cc:648] HybridClock initialized: now 1786008409017366 us; error 57 us; skew 500 ppm
I20260806 09:26:49.017858 9457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:26:49.020689 9457 webserver.cc:460] Webserver started at http://127.16.86.190:36209/ using document root <none> and password file <none>
I20260806 09:26:49.021471 9457 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:49.021613 9457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:49.021968 9457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:49.025312 9457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "dcfdf6f10bf241bba03344d84c778003"
format_stamp: "Formatted at 2026-08-06 09:26:49 on dist-test-slave-h3ft"
server_key: "963e8268fc67be3d76b74f5dd664d8ac"
server_key_iv: "c6cf09cd563c60872b5bd6f6eaa75a4d"
server_key_version: "encryptionkey@0"
I20260806 09:26:49.026207 9457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "dcfdf6f10bf241bba03344d84c778003"
format_stamp: "Formatted at 2026-08-06 09:26:49 on dist-test-slave-h3ft"
server_key: "963e8268fc67be3d76b74f5dd664d8ac"
server_key_iv: "c6cf09cd563c60872b5bd6f6eaa75a4d"
server_key_version: "encryptionkey@0"
I20260806 09:26:49.032420 9457 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260806 09:26:49.036481 9479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:49.038259 9457 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:26:49.038515 9457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "dcfdf6f10bf241bba03344d84c778003"
format_stamp: "Formatted at 2026-08-06 09:26:49 on dist-test-slave-h3ft"
server_key: "963e8268fc67be3d76b74f5dd664d8ac"
server_key_iv: "c6cf09cd563c60872b5bd6f6eaa75a4d"
server_key_version: "encryptionkey@0"
I20260806 09:26:49.038779 9457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:49.058740 9457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:49.059751 9457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:49.060079 9457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:49.083829 9457 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:38433
I20260806 09:26:49.083966 9548 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:38433 every 8 connection(s)
I20260806 09:26:49.085407 9457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:26:49.090770 9549 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:49.092986 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 9457
I20260806 09:26:49.093243 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:26:49.093767 16730 external_mini_cluster.cc:1428] Setting key bc14a842d64d94175c9d6577fc4ef286
I20260806 09:26:49.108971 9549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003: Bootstrap starting.
I20260806 09:26:49.111759 9552 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:49.096302 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35654 (local address 127.16.86.190:38433)
0806 09:26:49.098324 (+ 2022us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:49.098340 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:49.098402 (+ 62us) server_negotiation.cc:407] Connection header received
0806 09:26:49.100718 (+ 2316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:49.100753 (+ 35us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:49.101413 (+ 660us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:49.101954 (+ 541us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:49.102653 (+ 699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:49.104279 (+ 1626us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:49.105874 (+ 1595us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:49.106762 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:49.107033 (+ 271us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:49.108710 (+ 1677us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:49.108720 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:49.108738 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:49.109481 (+ 743us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:49.109585 (+ 104us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:49.109612 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:49.109790 (+ 178us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:49.110060 (+ 270us) server_negotiation.cc:299] Negotiation successful
0806 09:26:49.110473 (+ 413us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1286,"thread_start_us":906,"threads_started":1}
I20260806 09:26:49.113999 9549 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:49.115505 9549 log.cc:824] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:49.119630 9549 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003: No bootstrap required, opened a new log
I20260806 09:26:49.128334 9549 raft_consensus.cc:348] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfdf6f10bf241bba03344d84c778003" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38433 } }
I20260806 09:26:49.128854 9549 raft_consensus.cc:374] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:49.128974 9549 raft_consensus.cc:729] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcfdf6f10bf241bba03344d84c778003, State: Initialized, Role: FOLLOWER
I20260806 09:26:49.129709 9549 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfdf6f10bf241bba03344d84c778003" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38433 } }
I20260806 09:26:49.129989 9549 raft_consensus.cc:388] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:26:49.130172 9549 raft_consensus.cc:482] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:26:49.130395 9549 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:49.133497 9549 raft_consensus.cc:504] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfdf6f10bf241bba03344d84c778003" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38433 } }
I20260806 09:26:49.134215 9549 leader_election.cc:302] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcfdf6f10bf241bba03344d84c778003; no voters:
I20260806 09:26:49.134979 9549 leader_election.cc:288] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:26:49.135310 9554 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:26:49.136469 9554 raft_consensus.cc:686] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [term 1 LEADER]: Becoming Leader. State: Replica: dcfdf6f10bf241bba03344d84c778003, State: Running, Role: LEADER
I20260806 09:26:49.137192 9554 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfdf6f10bf241bba03344d84c778003" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38433 } }
I20260806 09:26:49.137916 9549 sys_catalog.cc:564] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:26:49.140625 9557 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcfdf6f10bf241bba03344d84c778003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfdf6f10bf241bba03344d84c778003" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38433 } } }
I20260806 09:26:49.141218 9557 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [sys.catalog]: This master's current role is: LEADER
I20260806 09:26:49.140625 9559 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcfdf6f10bf241bba03344d84c778003. Latest consensus state: current_term: 1 leader_uuid: "dcfdf6f10bf241bba03344d84c778003" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcfdf6f10bf241bba03344d84c778003" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38433 } } }
I20260806 09:26:49.141742 9559 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003 [sys.catalog]: This master's current role is: LEADER
I20260806 09:26:49.149273 9566 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:26:49.154983 9566 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:26:49.167689 9566 catalog_manager.cc:1347] Generated new cluster ID: 71b355f2143244b78a9e4f5eadbe1c89
I20260806 09:26:49.167845 9566 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:26:49.194810 9566 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:26:49.196259 9566 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:26:49.214267 9566 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003: Generated new TSK 0
I20260806 09:26:49.215338 9566 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:26:49.226833 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38433
--builtin_ntp_servers=127.16.86.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:26:49.593048 9589 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:49.593362 9589 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:49.593452 9589 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:49.600818 9589 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:49.600939 9589 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:49.601051 9589 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:26:49.610584 9589 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:49.612106 9589 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:49.614080 9589 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:49.629753 9597 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:49.629837 9595 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:49.630677 9599 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:49.631431 9589 server_base.cc:1034] running on GCE node
I20260806 09:26:49.632465 9589 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:49.633922 9589 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:49.635169 9589 hybrid_clock.cc:648] HybridClock initialized: now 1786008409635101 us; error 47 us; skew 500 ppm
I20260806 09:26:49.635641 9589 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:26:49.638795 9589 webserver.cc:460] Webserver started at http://127.16.86.129:41423/ using document root <none> and password file <none>
I20260806 09:26:49.639875 9589 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:49.640036 9589 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:49.640460 9589 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:49.644471 9589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "8953505ce122429fb28311991780b43c"
format_stamp: "Formatted at 2026-08-06 09:26:49 on dist-test-slave-h3ft"
server_key: "c0347dd6b7aece0ecdf4e15423856587"
server_key_iv: "205e1615894208dd51f14f13e95c08e6"
server_key_version: "encryptionkey@0"
I20260806 09:26:49.645517 9589 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "8953505ce122429fb28311991780b43c"
format_stamp: "Formatted at 2026-08-06 09:26:49 on dist-test-slave-h3ft"
server_key: "c0347dd6b7aece0ecdf4e15423856587"
server_key_iv: "205e1615894208dd51f14f13e95c08e6"
server_key_version: "encryptionkey@0"
I20260806 09:26:49.652267 9589 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260806 09:26:49.657068 9609 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:49.659122 9589 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260806 09:26:49.659366 9589 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "8953505ce122429fb28311991780b43c"
format_stamp: "Formatted at 2026-08-06 09:26:49 on dist-test-slave-h3ft"
server_key: "c0347dd6b7aece0ecdf4e15423856587"
server_key_iv: "205e1615894208dd51f14f13e95c08e6"
server_key_version: "encryptionkey@0"
I20260806 09:26:49.659675 9589 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:49.705895 9589 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:49.707099 9589 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:49.707496 9589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:49.708911 9589 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:49.710959 9589 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:49.711129 9589 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260806 09:26:49.711299 9589 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:49.711404 9589 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:49.755635 9589 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:35437
I20260806 09:26:49.755704 9758 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:35437 every 8 connection(s)
I20260806 09:26:49.757407 9589 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:26:49.763482 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 9589
I20260806 09:26:49.763705 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:26:49.764180 16730 external_mini_cluster.cc:1428] Setting key ea1e57fc9d84e424e7decb7e09af4fad
I20260806 09:26:49.768123 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38433
--builtin_ntp_servers=127.16.86.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:26:49.774923 9763 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:49.760686 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:56757 (local address 127.16.86.190:38433)
0806 09:26:49.761307 (+ 621us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:49.761312 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:49.762609 (+ 1297us) server_negotiation.cc:407] Connection header received
0806 09:26:49.765186 (+ 2577us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:49.765193 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:49.765304 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:49.765474 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:49.767339 (+ 1865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:49.768525 (+ 1186us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:49.770566 (+ 2041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:49.771312 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:49.771478 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:49.773729 (+ 2251us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:49.773738 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:49.773746 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:49.773775 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:49.773835 (+ 60us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:49.773855 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:49.773969 (+ 114us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:49.774321 (+ 352us) server_negotiation.cc:299] Negotiation successful
0806 09:26:49.774444 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":358,"threads_started":1}
I20260806 09:26:49.776430 9764 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:49.761182 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:38433 (local address 127.16.86.129:56757)
0806 09:26:49.762389 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:49.762443 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:49.764721 (+ 2278us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:49.765823 (+ 1102us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:49.765891 (+ 68us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:49.766035 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:49.767119 (+ 1084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:49.767143 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:49.768822 (+ 1679us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:49.768827 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:49.770381 (+ 1554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:49.770397 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:49.771581 (+ 1184us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:49.771586 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:49.771977 (+ 391us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:49.773238 (+ 1261us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:26:49.773241 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:49.773318 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:26:49.773445 (+ 127us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:49.774004 (+ 559us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:49.774019 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:49.774028 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 09:26:49.774283 (+ 255us) client_negotiation.cc:240] Negotiation successful
0806 09:26:49.774727 (+ 444us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":581,"thread_start_us":355,"threads_started":1}
I20260806 09:26:49.779141 9761 heartbeater.cc:347] Connected to a master server at 127.16.86.190:38433
I20260806 09:26:49.779582 9761 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:49.780632 9761 heartbeater.cc:511] Master 127.16.86.190:38433 requested a full tablet report, sending...
I20260806 09:26:49.783725 9505 ts_manager.cc:194] Registered new tserver with Master: 8953505ce122429fb28311991780b43c (127.16.86.129:35437)
I20260806 09:26:49.785470 9505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.86.129:56757
W20260806 09:26:50.205892 9765 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:50.206403 9765 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:50.206576 9765 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:50.215534 9765 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:50.215727 9765 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:50.215929 9765 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:26:50.226931 9765 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:50.228968 9765 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:50.231448 9765 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:50.249032 9772 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:50.249370 9771 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:50.249337 9775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:50.250717 9765 server_base.cc:1034] running on GCE node
I20260806 09:26:50.251654 9765 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:50.252948 9765 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:50.254279 9765 hybrid_clock.cc:648] HybridClock initialized: now 1786008410254204 us; error 60 us; skew 500 ppm
I20260806 09:26:50.254739 9765 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:26:50.258008 9765 webserver.cc:460] Webserver started at http://127.16.86.130:34195/ using document root <none> and password file <none>
I20260806 09:26:50.258874 9765 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:50.259045 9765 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:50.259445 9765 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:50.262779 9765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "8a8fe2e60db6489abb3962b3818c3e22"
format_stamp: "Formatted at 2026-08-06 09:26:50 on dist-test-slave-h3ft"
server_key: "d9c11c56895dca2f65542df3dd879f7a"
server_key_iv: "a310f6da14972c2c5918f7afe3422f70"
server_key_version: "encryptionkey@0"
I20260806 09:26:50.263811 9765 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "8a8fe2e60db6489abb3962b3818c3e22"
format_stamp: "Formatted at 2026-08-06 09:26:50 on dist-test-slave-h3ft"
server_key: "d9c11c56895dca2f65542df3dd879f7a"
server_key_iv: "a310f6da14972c2c5918f7afe3422f70"
server_key_version: "encryptionkey@0"
I20260806 09:26:50.270416 9765 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:26:50.274964 9782 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:50.276822 9765 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:26:50.277055 9765 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "8a8fe2e60db6489abb3962b3818c3e22"
format_stamp: "Formatted at 2026-08-06 09:26:50 on dist-test-slave-h3ft"
server_key: "d9c11c56895dca2f65542df3dd879f7a"
server_key_iv: "a310f6da14972c2c5918f7afe3422f70"
server_key_version: "encryptionkey@0"
I20260806 09:26:50.277338 9765 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:50.307968 9765 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:50.308947 9765 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:50.309337 9765 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:50.310578 9765 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:50.312433 9765 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:50.312608 9765 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:50.312775 9765 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:50.312876 9765 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:50.354585 9765 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:45809
I20260806 09:26:50.354702 9935 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:45809 every 8 connection(s)
I20260806 09:26:50.356305 9765 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:26:50.363199 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 9765
I20260806 09:26:50.363441 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:26:50.364056 16730 external_mini_cluster.cc:1428] Setting key f3eb367ca377e0054f7e07d9f7adb550
I20260806 09:26:50.368039 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38433
--builtin_ntp_servers=127.16.86.148:33373
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:26:50.374941 9940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:50.360170 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:41125 (local address 127.16.86.190:38433)
0806 09:26:50.360856 (+ 686us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:50.360863 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:50.362482 (+ 1619us) server_negotiation.cc:407] Connection header received
0806 09:26:50.365630 (+ 3148us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:50.365637 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:50.365839 (+ 202us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:50.366021 (+ 182us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:50.367871 (+ 1850us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:50.368883 (+ 1012us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:50.370940 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:50.371545 (+ 605us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:50.371740 (+ 195us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:50.373726 (+ 1986us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:50.373733 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:50.373737 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:50.373762 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:50.373821 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:50.373837 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:50.373971 (+ 134us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:50.374282 (+ 311us) server_negotiation.cc:299] Negotiation successful
0806 09:26:50.374438 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":325,"threads_started":1}
I20260806 09:26:50.376427 9941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:50.360869 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:38433 (local address 127.16.86.130:41125)
0806 09:26:50.362211 (+ 1342us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:50.362275 (+ 64us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:50.365163 (+ 2888us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:50.366380 (+ 1217us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:50.366392 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:50.366571 (+ 179us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:50.367625 (+ 1054us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:50.367658 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:50.369091 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:50.369099 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:50.370719 (+ 1620us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:50.370736 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:50.371877 (+ 1141us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:50.371887 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:50.372201 (+ 314us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:50.373266 (+ 1065us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:26:50.373269 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:50.373334 (+ 65us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:26:50.373447 (+ 113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:50.373969 (+ 522us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:50.373985 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:50.373998 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 09:26:50.374268 (+ 270us) client_negotiation.cc:240] Negotiation successful
0806 09:26:50.374835 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":701,"thread_start_us":413,"threads_started":1}
I20260806 09:26:50.378782 9937 heartbeater.cc:347] Connected to a master server at 127.16.86.190:38433
I20260806 09:26:50.379341 9937 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:50.380574 9937 heartbeater.cc:511] Master 127.16.86.190:38433 requested a full tablet report, sending...
I20260806 09:26:50.382699 9505 ts_manager.cc:194] Registered new tserver with Master: 8a8fe2e60db6489abb3962b3818c3e22 (127.16.86.130:45809)
I20260806 09:26:50.383719 9505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.86.130:41125
W20260806 09:26:50.780783 9942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:50.781158 9942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:50.781328 9942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:50.789103 9942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:50.789258 9942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
I20260806 09:26:50.789296 9761 heartbeater.cc:503] Master 127.16.86.190:38433 was elected leader, sending a full tablet report...
W20260806 09:26:50.789461 9942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:26:50.799227 9942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33373
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38433
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:50.800793 9942 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:50.802762 9942 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:50.820068 9949 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:50.820048 9950 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:50.820099 9953 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:50.821758 9942 server_base.cc:1034] running on GCE node
I20260806 09:26:50.822755 9942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:50.824229 9942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:50.825592 9942 hybrid_clock.cc:648] HybridClock initialized: now 1786008410825421 us; error 108 us; skew 500 ppm
I20260806 09:26:50.826078 9942 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:26:50.829506 9942 webserver.cc:460] Webserver started at http://127.16.86.131:36611/ using document root <none> and password file <none>
I20260806 09:26:50.830407 9942 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:50.830510 9942 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:50.830821 9942 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:50.834447 9942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "e21043f51cd04d0caf59aa8df5f40c12"
format_stamp: "Formatted at 2026-08-06 09:26:50 on dist-test-slave-h3ft"
server_key: "5f8f5c769f54c5bf5a36d8a8e503b8cf"
server_key_iv: "794778fea06acf4de90d2b46f5949d02"
server_key_version: "encryptionkey@0"
I20260806 09:26:50.835448 9942 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "e21043f51cd04d0caf59aa8df5f40c12"
format_stamp: "Formatted at 2026-08-06 09:26:50 on dist-test-slave-h3ft"
server_key: "5f8f5c769f54c5bf5a36d8a8e503b8cf"
server_key_iv: "794778fea06acf4de90d2b46f5949d02"
server_key_version: "encryptionkey@0"
I20260806 09:26:50.844096 9942 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260806 09:26:50.850030 9962 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:50.852461 9942 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260806 09:26:50.852748 9942 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "e21043f51cd04d0caf59aa8df5f40c12"
format_stamp: "Formatted at 2026-08-06 09:26:50 on dist-test-slave-h3ft"
server_key: "5f8f5c769f54c5bf5a36d8a8e503b8cf"
server_key_iv: "794778fea06acf4de90d2b46f5949d02"
server_key_version: "encryptionkey@0"
I20260806 09:26:50.853060 9942 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:50.884542 9942 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:50.885900 9942 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:50.886324 9942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:50.887758 9942 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:50.889650 9942 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:50.889755 9942 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:50.889941 9942 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:50.890023 9942 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:50.932432 9942 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:35261
I20260806 09:26:50.932698 10114 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:35261 every 8 connection(s)
I20260806 09:26:50.934176 9942 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:26:50.941447 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 9942
I20260806 09:26:50.941700 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:26:50.942255 16730 external_mini_cluster.cc:1428] Setting key 75a5765cb57eef95701cf282cf2992e5
I20260806 09:26:50.951561 10118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:50.937739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:45427 (local address 127.16.86.190:38433)
0806 09:26:50.938349 (+ 610us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:50.938355 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:50.940104 (+ 1749us) server_negotiation.cc:407] Connection header received
0806 09:26:50.942296 (+ 2192us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:50.942305 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:50.942419 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:50.942618 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:50.944606 (+ 1988us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:50.945635 (+ 1029us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:50.947974 (+ 2339us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:50.948446 (+ 472us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:50.948560 (+ 114us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:50.950406 (+ 1846us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:50.950412 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:50.950415 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:50.950436 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:50.950483 (+ 47us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:50.950496 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:50.950573 (+ 77us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:50.950893 (+ 320us) server_negotiation.cc:299] Negotiation successful
0806 09:26:50.950987 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":322,"threads_started":1}
I20260806 09:26:50.953159 10119 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:50.938463 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:38433 (local address 127.16.86.131:45427)
0806 09:26:50.939835 (+ 1372us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:50.939901 (+ 66us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:50.941947 (+ 2046us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:50.943073 (+ 1126us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:50.943088 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:50.943266 (+ 178us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:50.944340 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:50.944363 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:50.945933 (+ 1570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:50.945939 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:50.947792 (+ 1853us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:50.947816 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:50.948711 (+ 895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:50.948716 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:50.948964 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:50.949949 (+ 985us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:26:50.949953 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:50.950012 (+ 59us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:26:50.950143 (+ 131us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:50.950631 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:50.950641 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:50.950649 (+ 8us) client_negotiation.cc:769] Sending connection context
0806 09:26:50.950839 (+ 190us) client_negotiation.cc:240] Negotiation successful
0806 09:26:50.951483 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":717,"thread_start_us":458,"threads_started":1}
I20260806 09:26:50.955273 10115 heartbeater.cc:347] Connected to a master server at 127.16.86.190:38433
I20260806 09:26:50.955749 10115 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:50.956833 10115 heartbeater.cc:511] Master 127.16.86.190:38433 requested a full tablet report, sending...
I20260806 09:26:50.959100 9505 ts_manager.cc:194] Registered new tserver with Master: e21043f51cd04d0caf59aa8df5f40c12 (127.16.86.131:35261)
I20260806 09:26:50.960286 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:26:50.960392 9505 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.86.131:45427
I20260806 09:26:50.973894 10118 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:50.966888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35656 (local address 127.16.86.190:38433)
0806 09:26:50.967271 (+ 383us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:50.967281 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:50.967723 (+ 442us) server_negotiation.cc:407] Connection header received
0806 09:26:50.967967 (+ 244us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:50.967981 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:50.968052 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:50.968191 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:50.968816 (+ 625us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:50.969640 (+ 824us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:50.971191 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:50.971646 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:50.971803 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:50.973128 (+ 1325us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:50.973134 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:50.973137 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:50.973156 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:50.973188 (+ 32us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:50.973200 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:50.973312 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:50.973474 (+ 162us) server_negotiation.cc:299] Negotiation successful
0806 09:26:50.973573 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":165}
I20260806 09:26:50.978823 9505 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:35656:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:26:50.986438 9505 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:26:51.014385 10130 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.002515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35437 (local address 127.0.0.1:56102)
0806 09:26:51.005071 (+ 2556us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.005128 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:51.005340 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:51.006106 (+ 766us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:51.006113 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:51.006144 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:51.006562 (+ 418us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:51.006573 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.008510 (+ 1937us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.008515 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:51.011147 (+ 2632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:51.011167 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.013280 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.013286 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:51.013563 (+ 277us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.013642 (+ 79us) client_negotiation.cc:769] Sending connection context
0806 09:26:51.013764 (+ 122us) client_negotiation.cc:240] Negotiation successful
0806 09:26:51.013968 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1507,"mutex_wait_us":4,"spinlock_wait_cycles":5376,"thread_start_us":1232,"threads_started":1}
I20260806 09:26:51.015434 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.002804 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56102 (local address 127.16.86.129:35437)
0806 09:26:51.003864 (+ 1060us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:51.003876 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:51.005566 (+ 1690us) server_negotiation.cc:407] Connection header received
0806 09:26:51.005667 (+ 101us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:51.005674 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:51.005887 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:51.006105 (+ 218us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:51.006764 (+ 659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.008357 (+ 1593us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.011429 (+ 3072us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.013027 (+ 1598us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.014050 (+ 1023us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.014523 (+ 473us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:51.014640 (+ 117us) server_negotiation.cc:299] Negotiation successful
0806 09:26:51.014900 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":776,"thread_start_us":488,"threads_started":1}
I20260806 09:26:51.019254 10135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.004253 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:45809 (local address 127.0.0.1:46116)
0806 09:26:51.007980 (+ 3727us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.008000 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:51.008114 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:51.009074 (+ 960us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:51.009079 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:51.009107 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:51.009541 (+ 434us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:51.009554 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.011434 (+ 1880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.011439 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:51.015289 (+ 3850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:51.015315 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.018227 (+ 2912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.018234 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:51.018456 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.018499 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 09:26:51.018591 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 09:26:51.018782 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3600,"mutex_wait_us":1701,"thread_start_us":3450,"threads_started":1}
I20260806 09:26:51.019629 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.004848 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46116 (local address 127.16.86.130:45809)
0806 09:26:51.007525 (+ 2677us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:51.007534 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:51.008225 (+ 691us) server_negotiation.cc:407] Connection header received
0806 09:26:51.008339 (+ 114us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:51.008346 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:51.008580 (+ 234us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:51.008803 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:51.009772 (+ 969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.011280 (+ 1508us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.015636 (+ 4356us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.017281 (+ 1645us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.017781 (+ 500us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.018080 (+ 299us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:51.018983 (+ 903us) server_negotiation.cc:299] Negotiation successful
0806 09:26:51.019227 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2476,"thread_start_us":2328,"threads_started":1}
I20260806 09:26:51.020287 9671 tablet_service.cc:1467] Processing CreateTablet for tablet 361ad96175ce481e9d9d662c6e995685 (DEFAULT_TABLE table=test-table [id=6928105c875c4744a9e492da784ac8a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:51.022127 9671 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361ad96175ce481e9d9d662c6e995685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:51.022135 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.002754 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:35261 (local address 127.0.0.1:50644)
0806 09:26:51.005071 (+ 2317us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.005128 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:51.005340 (+ 212us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:51.014650 (+ 9310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:51.014656 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:51.014683 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:51.015184 (+ 501us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:51.015200 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.017010 (+ 1810us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.017019 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:51.019214 (+ 2195us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:51.019233 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.021124 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.021133 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:51.021340 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.021376 (+ 36us) client_negotiation.cc:769] Sending connection context
0806 09:26:51.021471 (+ 95us) client_negotiation.cc:240] Negotiation successful
0806 09:26:51.021635 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2167,"spinlock_wait_cycles":7424,"thread_start_us":2033,"threads_started":1}
I20260806 09:26:51.022173 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.003658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50644 (local address 127.16.86.131:35261)
0806 09:26:51.013977 (+ 10319us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:51.013986 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:51.014024 (+ 38us) server_negotiation.cc:407] Connection header received
0806 09:26:51.014128 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:51.014137 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:51.014406 (+ 269us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:51.015439 (+ 1033us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:51.015755 (+ 316us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.016765 (+ 1010us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.019466 (+ 2701us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.020872 (+ 1406us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.021036 (+ 164us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.021353 (+ 317us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:51.021566 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 09:26:51.021771 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10097,"thread_start_us":9915,"threads_started":1}
I20260806 09:26:51.024510 9850 tablet_service.cc:1467] Processing CreateTablet for tablet 361ad96175ce481e9d9d662c6e995685 (DEFAULT_TABLE table=test-table [id=6928105c875c4744a9e492da784ac8a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:51.026393 9850 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361ad96175ce481e9d9d662c6e995685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:51.026809 10026 tablet_service.cc:1467] Processing CreateTablet for tablet 361ad96175ce481e9d9d662c6e995685 (DEFAULT_TABLE table=test-table [id=6928105c875c4744a9e492da784ac8a4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:51.028551 10026 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 361ad96175ce481e9d9d662c6e995685. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:51.035282 10137 tablet_bootstrap.cc:492] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c: Bootstrap starting.
I20260806 09:26:51.039558 10137 tablet_bootstrap.cc:654] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:51.040707 10137 log.cc:824] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:51.043819 10137 tablet_bootstrap.cc:492] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c: No bootstrap required, opened a new log
I20260806 09:26:51.044215 10137 ts_tablet_manager.cc:1397] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20260806 09:26:51.044363 10139 tablet_bootstrap.cc:492] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22: Bootstrap starting.
I20260806 09:26:51.047683 10142 tablet_bootstrap.cc:492] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12: Bootstrap starting.
I20260806 09:26:51.048600 10139 tablet_bootstrap.cc:654] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:51.049908 10139 log.cc:824] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:51.051772 10142 tablet_bootstrap.cc:654] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:51.053066 10142 log.cc:824] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:51.053809 10139 tablet_bootstrap.cc:492] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22: No bootstrap required, opened a new log
I20260806 09:26:51.054224 10139 ts_tablet_manager.cc:1397] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.008s
I20260806 09:26:51.056200 10137 raft_consensus.cc:348] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953505ce122429fb28311991780b43c" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35437 } } peers { permanent_uuid: "e21043f51cd04d0caf59aa8df5f40c12" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 35261 } } peers { permanent_uuid: "8a8fe2e60db6489abb3962b3818c3e22" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45809 } }
I20260806 09:26:51.056879 10137 raft_consensus.cc:374] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:51.056979 10142 tablet_bootstrap.cc:492] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12: No bootstrap required, opened a new log
I20260806 09:26:51.057029 10137 raft_consensus.cc:729] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8953505ce122429fb28311991780b43c, State: Initialized, Role: FOLLOWER
I20260806 09:26:51.057452 10142 ts_tablet_manager.cc:1397] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.006s
I20260806 09:26:51.057950 10137 consensus_queue.cc:260] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953505ce122429fb28311991780b43c" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35437 } } peers { permanent_uuid: "e21043f51cd04d0caf59aa8df5f40c12" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 35261 } } peers { permanent_uuid: "8a8fe2e60db6489abb3962b3818c3e22" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45809 } }
I20260806 09:26:51.060258 10137 ts_tablet_manager.cc:1428] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c: Time spent starting tablet: real 0.016s user 0.014s sys 0.000s
I20260806 09:26:51.065290 10139 raft_consensus.cc:348] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953505ce122429fb28311991780b43c" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35437 } } peers { permanent_uuid: "e21043f51cd04d0caf59aa8df5f40c12" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 35261 } } peers { permanent_uuid: "8a8fe2e60db6489abb3962b3818c3e22" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45809 } }
I20260806 09:26:51.065764 10139 raft_consensus.cc:374] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:51.065881 10139 raft_consensus.cc:729] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8a8fe2e60db6489abb3962b3818c3e22, State: Initialized, Role: FOLLOWER
I20260806 09:26:51.066587 10139 consensus_queue.cc:260] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953505ce122429fb28311991780b43c" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35437 } } peers { permanent_uuid: "e21043f51cd04d0caf59aa8df5f40c12" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 35261 } } peers { permanent_uuid: "8a8fe2e60db6489abb3962b3818c3e22" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45809 } }
I20260806 09:26:51.068068 9937 heartbeater.cc:503] Master 127.16.86.190:38433 was elected leader, sending a full tablet report...
I20260806 09:26:51.068763 10139 ts_tablet_manager.cc:1428] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22: Time spent starting tablet: real 0.014s user 0.013s sys 0.000s
I20260806 09:26:51.069969 10142 raft_consensus.cc:348] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953505ce122429fb28311991780b43c" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35437 } } peers { permanent_uuid: "e21043f51cd04d0caf59aa8df5f40c12" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 35261 } } peers { permanent_uuid: "8a8fe2e60db6489abb3962b3818c3e22" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45809 } }
I20260806 09:26:51.070436 10142 raft_consensus.cc:374] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:51.070601 10142 raft_consensus.cc:729] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e21043f51cd04d0caf59aa8df5f40c12, State: Initialized, Role: FOLLOWER
I20260806 09:26:51.071368 10142 consensus_queue.cc:260] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953505ce122429fb28311991780b43c" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35437 } } peers { permanent_uuid: "e21043f51cd04d0caf59aa8df5f40c12" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 35261 } } peers { permanent_uuid: "8a8fe2e60db6489abb3962b3818c3e22" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45809 } }
I20260806 09:26:51.072911 10115 heartbeater.cc:503] Master 127.16.86.190:38433 was elected leader, sending a full tablet report...
I20260806 09:26:51.073396 10142 ts_tablet_manager.cc:1428] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12: Time spent starting tablet: real 0.016s user 0.010s sys 0.004s
W20260806 09:26:51.109138 9938 tablet.cc:2369] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:26:51.186517 10116 tablet.cc:2369] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:26:51.196986 10147 raft_consensus.cc:482] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:26:51.197453 10147 raft_consensus.cc:504] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953505ce122429fb28311991780b43c" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35437 } } peers { permanent_uuid: "e21043f51cd04d0caf59aa8df5f40c12" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 35261 } } peers { permanent_uuid: "8a8fe2e60db6489abb3962b3818c3e22" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45809 } }
I20260806 09:26:51.199978 10147 leader_election.cc:288] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8953505ce122429fb28311991780b43c (127.16.86.129:35437), e21043f51cd04d0caf59aa8df5f40c12 (127.16.86.131:35261)
I20260806 09:26:51.208799 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.200712 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:49845 (local address 127.16.86.131:35261)
0806 09:26:51.200970 (+ 258us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:51.200977 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:51.201802 (+ 825us) server_negotiation.cc:407] Connection header received
0806 09:26:51.202171 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:51.202179 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:51.202312 (+ 133us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:51.202419 (+ 107us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:51.203657 (+ 1238us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.204442 (+ 785us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.206708 (+ 2266us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.207653 (+ 945us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.207897 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.207978 (+ 81us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:51.208340 (+ 362us) server_negotiation.cc:299] Negotiation successful
0806 09:26:51.208474 (+ 134us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 09:26:51.208869 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.200524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:44139 (local address 127.16.86.129:35437)
0806 09:26:51.200830 (+ 306us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:51.200839 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:51.201556 (+ 717us) server_negotiation.cc:407] Connection header received
0806 09:26:51.201983 (+ 427us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:51.201992 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:51.202118 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:51.202276 (+ 158us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:51.203310 (+ 1034us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.204268 (+ 958us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.206608 (+ 2340us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.207671 (+ 1063us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.207898 (+ 227us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.208001 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:51.208395 (+ 394us) server_negotiation.cc:299] Negotiation successful
0806 09:26:51.208547 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:26:51.209071 10149 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.200258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35437 (local address 127.16.86.130:44139)
0806 09:26:51.201367 (+ 1109us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.201406 (+ 39us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:51.201683 (+ 277us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:51.202337 (+ 654us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:51.202345 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:51.202384 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:51.202981 (+ 597us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:51.203065 (+ 84us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.204444 (+ 1379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.204450 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:51.206357 (+ 1907us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:51.206375 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.207982 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.207992 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:51.208197 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.208226 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 09:26:51.208362 (+ 136us) client_negotiation.cc:240] Negotiation successful
0806 09:26:51.208545 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":903,"mutex_wait_us":6,"spinlock_wait_cycles":32512,"thread_start_us":597,"threads_started":1}
I20260806 09:26:51.209205 10150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.200677 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:35261 (local address 127.16.86.130:49845)
0806 09:26:51.201662 (+ 985us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.201680 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:51.201907 (+ 227us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:51.202538 (+ 631us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:51.202543 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:51.202632 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:51.203340 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:51.203357 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.204654 (+ 1297us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.204661 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:51.206452 (+ 1791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:51.206467 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.207910 (+ 1443us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.207917 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:51.208147 (+ 230us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.208177 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 09:26:51.208308 (+ 131us) client_negotiation.cc:240] Negotiation successful
0806 09:26:51.208474 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":750,"mutex_wait_us":6,"spinlock_wait_cycles":4864,"thread_start_us":485,"threads_started":1}
I20260806 09:26:51.210753 10053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361ad96175ce481e9d9d662c6e995685" candidate_uuid: "8a8fe2e60db6489abb3962b3818c3e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e21043f51cd04d0caf59aa8df5f40c12" is_pre_election: true
I20260806 09:26:51.210750 9699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361ad96175ce481e9d9d662c6e995685" candidate_uuid: "8a8fe2e60db6489abb3962b3818c3e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8953505ce122429fb28311991780b43c" is_pre_election: true
I20260806 09:26:51.211593 9699 raft_consensus.cc:2454] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a8fe2e60db6489abb3962b3818c3e22 in term 0.
I20260806 09:26:51.211603 10053 raft_consensus.cc:2454] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8a8fe2e60db6489abb3962b3818c3e22 in term 0.
I20260806 09:26:51.212560 9790 leader_election.cc:302] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8a8fe2e60db6489abb3962b3818c3e22, e21043f51cd04d0caf59aa8df5f40c12; no voters:
I20260806 09:26:51.213099 10147 raft_consensus.cc:2781] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:26:51.213284 10147 raft_consensus.cc:482] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:26:51.213394 10147 raft_consensus.cc:3037] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:51.215952 10147 raft_consensus.cc:504] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953505ce122429fb28311991780b43c" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35437 } } peers { permanent_uuid: "e21043f51cd04d0caf59aa8df5f40c12" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 35261 } } peers { permanent_uuid: "8a8fe2e60db6489abb3962b3818c3e22" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45809 } }
I20260806 09:26:51.216655 10147 leader_election.cc:288] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [CANDIDATE]: Term 1 election: Requested vote from peers 8953505ce122429fb28311991780b43c (127.16.86.129:35437), e21043f51cd04d0caf59aa8df5f40c12 (127.16.86.131:35261)
I20260806 09:26:51.217370 10053 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361ad96175ce481e9d9d662c6e995685" candidate_uuid: "8a8fe2e60db6489abb3962b3818c3e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e21043f51cd04d0caf59aa8df5f40c12"
I20260806 09:26:51.217398 9699 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "361ad96175ce481e9d9d662c6e995685" candidate_uuid: "8a8fe2e60db6489abb3962b3818c3e22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8953505ce122429fb28311991780b43c"
I20260806 09:26:51.217658 10053 raft_consensus.cc:3037] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:51.217676 9699 raft_consensus.cc:3037] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:51.220122 9699 raft_consensus.cc:2454] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a8fe2e60db6489abb3962b3818c3e22 in term 1.
I20260806 09:26:51.220532 10053 raft_consensus.cc:2454] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8a8fe2e60db6489abb3962b3818c3e22 in term 1.
I20260806 09:26:51.220706 9791 leader_election.cc:302] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8953505ce122429fb28311991780b43c, 8a8fe2e60db6489abb3962b3818c3e22; no voters:
I20260806 09:26:51.221129 10147 raft_consensus.cc:2781] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:26:51.221733 10147 raft_consensus.cc:686] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 1 LEADER]: Becoming Leader. State: Replica: 8a8fe2e60db6489abb3962b3818c3e22, State: Running, Role: LEADER
I20260806 09:26:51.222257 10147 consensus_queue.cc:237] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953505ce122429fb28311991780b43c" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35437 } } peers { permanent_uuid: "e21043f51cd04d0caf59aa8df5f40c12" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 35261 } } peers { permanent_uuid: "8a8fe2e60db6489abb3962b3818c3e22" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45809 } }
I20260806 09:26:51.228319 9504 catalog_manager.cc:5495] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8a8fe2e60db6489abb3962b3818c3e22 (127.16.86.130). New cstate: current_term: 1 leader_uuid: "8a8fe2e60db6489abb3962b3818c3e22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8953505ce122429fb28311991780b43c" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35437 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e21043f51cd04d0caf59aa8df5f40c12" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 35261 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8a8fe2e60db6489abb3962b3818c3e22" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45809 } health_report { overall_health: HEALTHY } } }
W20260806 09:26:51.260996 9762 tablet.cc:2369] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:26:51.264747 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.257515 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46118 (local address 127.16.86.130:45809)
0806 09:26:51.257777 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:51.257782 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:51.257850 (+ 68us) server_negotiation.cc:407] Connection header received
0806 09:26:51.258115 (+ 265us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:51.258119 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:51.258228 (+ 109us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:51.258414 (+ 186us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:26:51.258925 (+ 511us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.259818 (+ 893us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.261682 (+ 1864us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.262215 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.262393 (+ 178us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.263132 (+ 739us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:26:51.263917 (+ 785us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:26:51.264117 (+ 200us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:51.264246 (+ 129us) server_negotiation.cc:299] Negotiation successful
0806 09:26:51.264357 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:26:51.275633 9699 raft_consensus.cc:1261] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c [term 1 FOLLOWER]: Refusing update from remote peer 8a8fe2e60db6489abb3962b3818c3e22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:26:51.275668 10053 raft_consensus.cc:1261] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12 [term 1 FOLLOWER]: Refusing update from remote peer 8a8fe2e60db6489abb3962b3818c3e22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:26:51.276924 10151 consensus_queue.cc:1035] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e21043f51cd04d0caf59aa8df5f40c12" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 35261 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:26:51.277354 10147 consensus_queue.cc:1035] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8953505ce122429fb28311991780b43c" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35437 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:26:51.293817 10157 mvcc.cc:205] Tried to move back new op lower bound from 7315490452570877952 to 7315490452370649088. Current Snapshot: MvccSnapshot[applied={T|T < 7315490452570877952}]
I20260806 09:26:51.294250 10155 mvcc.cc:205] Tried to move back new op lower bound from 7315490452570877952 to 7315490452370649088. Current Snapshot: MvccSnapshot[applied={T|T < 7315490452570877952}]
I20260806 09:26:51.300014 10159 mvcc.cc:205] Tried to move back new op lower bound from 7315490452570877952 to 7315490452370649088. Current Snapshot: MvccSnapshot[applied={T|T < 7315490452570877952 or (T in {7315490452570877952})}]
I20260806 09:26:51.314718 9505 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:35656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:26:51.315155 9505 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:35656:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:26:51.320026 9505 catalog_manager.cc:5782] T 00000000000000000000000000000000 P dcfdf6f10bf241bba03344d84c778003: Sending DeleteTablet for 3 replicas of tablet 361ad96175ce481e9d9d662c6e995685
I20260806 09:26:51.321709 10026 tablet_service.cc:1514] Processing DeleteTablet for tablet 361ad96175ce481e9d9d662c6e995685 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:26:51 UTC) from {username='slave'} at 127.0.0.1:50644
I20260806 09:26:51.322021 9850 tablet_service.cc:1514] Processing DeleteTablet for tablet 361ad96175ce481e9d9d662c6e995685 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:26:51 UTC) from {username='slave'} at 127.0.0.1:46116
I20260806 09:26:51.322682 9671 tablet_service.cc:1514] Processing DeleteTablet for tablet 361ad96175ce481e9d9d662c6e995685 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:26:51 UTC) from {username='slave'} at 127.0.0.1:56102
I20260806 09:26:51.323263 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 9589
I20260806 09:26:51.333709 10171 tablet_replica.cc:330] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12: stopping tablet replica
I20260806 09:26:51.334261 10171 raft_consensus.cc:2229] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:26:51.334841 10171 raft_consensus.cc:2258] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:26:51.335804 10132 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.324614 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56118 (local address 127.16.86.129:35437)
0806 09:26:51.324811 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:51.324818 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:51.324839 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:26:51.325086 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:51.325097 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:51.325238 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:51.325347 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:51.326079 (+ 732us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.327533 (+ 1454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.329945 (+ 2412us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.331080 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.331208 (+ 128us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.333241 (+ 2033us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:51.333258 (+ 17us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:51.333282 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:51.334441 (+ 1159us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:51.334520 (+ 79us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:51.334556 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:51.334708 (+ 152us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:51.335065 (+ 357us) server_negotiation.cc:299] Negotiation successful
0806 09:26:51.335235 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20260806 09:26:51.337210 9754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:51.337216 10172 tablet_replica.cc:330] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22: stopping tablet replica
I20260806 09:26:51.337711 10172 raft_consensus.cc:2229] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:26:51.338770 10172 raft_consensus.cc:2258] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:26:51.342208 10172 ts_tablet_manager.cc:1905] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:26:51.343987 10171 ts_tablet_manager.cc:1905] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:26:51.351210 10172 ts_tablet_manager.cc:1918] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:26:51.351590 10172 log.cc:1196] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/361ad96175ce481e9d9d662c6e995685
I20260806 09:26:51.352358 10172 ts_tablet_manager.cc:1939] T 361ad96175ce481e9d9d662c6e995685 P 8a8fe2e60db6489abb3962b3818c3e22: Deleting consensus metadata
I20260806 09:26:51.352643 10171 ts_tablet_manager.cc:1918] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:26:51.353003 10171 log.cc:1196] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/361ad96175ce481e9d9d662c6e995685
I20260806 09:26:51.353688 10171 ts_tablet_manager.cc:1939] T 361ad96175ce481e9d9d662c6e995685 P e21043f51cd04d0caf59aa8df5f40c12: Deleting consensus metadata
I20260806 09:26:51.356216 9485 catalog_manager.cc:4841] TS e21043f51cd04d0caf59aa8df5f40c12 (127.16.86.131:35261): tablet 361ad96175ce481e9d9d662c6e995685 (table test-table [id=6928105c875c4744a9e492da784ac8a4]) successfully deleted
I20260806 09:26:51.360756 9482 catalog_manager.cc:4841] TS 8a8fe2e60db6489abb3962b3818c3e22 (127.16.86.130:45809): tablet 361ad96175ce481e9d9d662c6e995685 (table test-table [id=6928105c875c4744a9e492da784ac8a4]) successfully deleted
I20260806 09:26:51.453403 10175 tablet_replica.cc:330] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c: stopping tablet replica
I20260806 09:26:51.453794 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 9589
I20260806 09:26:51.453954 10175 raft_consensus.cc:2229] T 361ad96175ce481e9d9d662c6e995685 P 8953505ce122429fb28311991780b43c [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:26:51.478750 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 9765
W20260806 09:26:51.478706 9487 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.86.129:35437 (error 108)
I20260806 09:26:51.481256 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.480532 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35437 (local address 127.0.0.1:56124)
0806 09:26:51.480810 (+ 278us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.481020 (+ 210us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20260806 09:26:51.481830 9487 catalog_manager.cc:4584] TS 8953505ce122429fb28311991780b43c (127.16.86.129:35437): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 361ad96175ce481e9d9d662c6e995685: Network error: Client connection negotiation failed: client connection to 127.16.86.129:35437: connect: Connection refused (error 111)
I20260806 09:26:51.487375 10136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.480226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46130 (local address 127.16.86.130:45809)
0806 09:26:51.480479 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:51.480486 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:51.480505 (+ 19us) server_negotiation.cc:407] Connection header received
0806 09:26:51.480588 (+ 83us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:51.480593 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:51.480730 (+ 137us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:51.480858 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:51.481333 (+ 475us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.482468 (+ 1135us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.484030 (+ 1562us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.484505 (+ 475us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.484664 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.485834 (+ 1170us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:51.485843 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:51.485856 (+ 13us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:51.486489 (+ 633us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:51.486542 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:51.486562 (+ 20us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:51.486706 (+ 144us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:51.486871 (+ 165us) server_negotiation.cc:299] Negotiation successful
0806 09:26:51.486967 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":111}
I20260806 09:26:51.488317 9930 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:51.536191 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.535431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35437 (local address 127.0.0.1:56140)
0806 09:26:51.535796 (+ 365us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.535948 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":139}
I20260806 09:26:51.537652 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.537197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35437 (local address 127.0.0.1:56150)
0806 09:26:51.537386 (+ 189us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.537484 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 09:26:51.598876 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.598060 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35437 (local address 127.0.0.1:56160)
0806 09:26:51.598464 (+ 404us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.598611 (+ 147us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":167}
I20260806 09:26:51.600570 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.600039 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35437 (local address 127.0.0.1:56164)
0806 09:26:51.600256 (+ 217us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.600391 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":92}
I20260806 09:26:51.613937 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 9765
I20260806 09:26:51.647320 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 9942
I20260806 09:26:51.655870 10134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.648356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50650 (local address 127.16.86.131:35261)
0806 09:26:51.648618 (+ 262us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:51.648624 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:51.648647 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:26:51.648777 (+ 130us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:51.648781 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:51.648903 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:51.649005 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:51.649578 (+ 573us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.650391 (+ 813us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.651923 (+ 1532us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:51.652343 (+ 420us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:51.652453 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:51.653742 (+ 1289us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:51.653754 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:51.653774 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:51.654605 (+ 831us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:51.654671 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:51.654703 (+ 32us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:51.654824 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:51.655285 (+ 461us) server_negotiation.cc:299] Negotiation successful
0806 09:26:51.655465 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 09:26:51.656900 10107 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:51.681928 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.681040 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35437 (local address 127.0.0.1:56176)
0806 09:26:51.681433 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.681614 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":133}
I20260806 09:26:51.683614 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.683054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35437 (local address 127.0.0.1:56192)
0806 09:26:51.683337 (+ 283us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.683443 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":159}
I20260806 09:26:51.762465 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 9942
I20260806 09:26:51.791599 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 9457
I20260806 09:26:51.793212 9544 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:51.909713 10131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:51.908889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35437 (local address 127.0.0.1:56194)
0806 09:26:51.909228 (+ 339us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:51.909380 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:35437: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":119}
I20260806 09:26:51.910360 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 9457
2026-08-06T09:26:51Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (3443 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:26:52 dist-test-slave-h3ft krb5kdc[10185](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:26:52 dist-test-slave-h3ft krb5kdc[10185](info): set up 2 sockets
Aug 06 09:26:52 dist-test-slave-h3ft krb5kdc[10185](info): commencing operation
krb5kdc: starting...
W20260806 09:26:53.295219 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.321s user 0.000s sys 0.065s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:26:53 dist-test-slave-h3ft krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:26:53Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:26:53Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:26:53.699761 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:39119
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:39119
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:26:54.097482 10199 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:54.097887 10199 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:54.097983 10199 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:54.105527 10199 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:26:54.105677 10199 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:54.105728 10199 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:54.105845 10199 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:26:54.105924 10199 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:26:54.116699 10199 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:39119
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:39119
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:54.118319 10199 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:54.119750 10199 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:54.133844 10209 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:54.133996 10212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:54.134446 10214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:54.135205 10199 server_base.cc:1034] running on GCE node
I20260806 09:26:54.136272 10199 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:54.137912 10199 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:54.139221 10199 hybrid_clock.cc:648] HybridClock initialized: now 1786008414139153 us; error 70 us; skew 500 ppm
I20260806 09:26:54.139784 10199 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:54 dist-test-slave-h3ft krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008414, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:54.147390 10199 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:26:54.149384 10199 webserver.cc:460] Webserver started at http://127.16.86.190:37529/ using document root <none> and password file <none>
I20260806 09:26:54.150136 10199 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:54.150252 10199 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:54.150529 10199 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:54.153832 10199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "9ab5f1b922984bba8f33573db3cc9b3f"
format_stamp: "Formatted at 2026-08-06 09:26:54 on dist-test-slave-h3ft"
server_key: "63be1eee81891e545569983f8846a505"
server_key_iv: "26ddbc26eec0d4f4d9ce1dec0606dee1"
server_key_version: "encryptionkey@0"
I20260806 09:26:54.154580 10199 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "9ab5f1b922984bba8f33573db3cc9b3f"
format_stamp: "Formatted at 2026-08-06 09:26:54 on dist-test-slave-h3ft"
server_key: "63be1eee81891e545569983f8846a505"
server_key_iv: "26ddbc26eec0d4f4d9ce1dec0606dee1"
server_key_version: "encryptionkey@0"
I20260806 09:26:54.160904 10199 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20260806 09:26:54.165541 10222 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:54.167483 10199 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:26:54.167719 10199 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "9ab5f1b922984bba8f33573db3cc9b3f"
format_stamp: "Formatted at 2026-08-06 09:26:54 on dist-test-slave-h3ft"
server_key: "63be1eee81891e545569983f8846a505"
server_key_iv: "26ddbc26eec0d4f4d9ce1dec0606dee1"
server_key_version: "encryptionkey@0"
I20260806 09:26:54.167919 10199 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:54.186290 10199 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:54.190958 10199 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:54.191246 10199 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:54.213593 10199 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:39119
I20260806 09:26:54.213665 10293 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:39119 every 8 connection(s)
I20260806 09:26:54.215301 10199 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:26:54.220718 10294 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:54.220839 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 10199
I20260806 09:26:54.221065 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:26:54.221760 16730 external_mini_cluster.cc:1428] Setting key 499434c4aba3347e7f43b215a26c8f2f
I20260806 09:26:54.240883 10294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f: Bootstrap starting.
Aug 06 09:26:54 dist-test-slave-h3ft krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:26:54.245697 10294 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:54.247267 10294 log.cc:824] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:54.251415 10294 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f: No bootstrap required, opened a new log
I20260806 09:26:54.252987 10299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:54.224374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35022 (local address 127.16.86.190:39119)
0806 09:26:54.225588 (+ 1214us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:54.225601 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:54.225657 (+ 56us) server_negotiation.cc:407] Connection header received
0806 09:26:54.227703 (+ 2046us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:54.227728 (+ 25us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:54.228190 (+ 462us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:54.228577 (+ 387us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:54.231192 (+ 2615us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:54.232916 (+ 1724us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:54.235445 (+ 2529us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:54.236356 (+ 911us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:54.236565 (+ 209us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:54.243302 (+ 6737us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:54.243315 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:54.243336 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:54.243400 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:54.247229 (+ 3829us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:54.248347 (+ 1118us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:54.248358 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:54.248367 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:54.248657 (+ 290us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:54.249390 (+ 733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:54.249397 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:54.249406 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:54.250156 (+ 750us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:54.250553 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:54.251178 (+ 625us) server_negotiation.cc:299] Negotiation successful
0806 09:26:54.251754 (+ 576us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":957,"thread_start_us":680,"threads_started":1}
I20260806 09:26:54.261790 10294 raft_consensus.cc:348] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab5f1b922984bba8f33573db3cc9b3f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 39119 } }
I20260806 09:26:54.262483 10294 raft_consensus.cc:374] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:54.262622 10294 raft_consensus.cc:729] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ab5f1b922984bba8f33573db3cc9b3f, State: Initialized, Role: FOLLOWER
I20260806 09:26:54.263572 10294 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab5f1b922984bba8f33573db3cc9b3f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 39119 } }
I20260806 09:26:54.263967 10294 raft_consensus.cc:388] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:26:54.264186 10294 raft_consensus.cc:482] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:26:54.264424 10294 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:54.267302 10294 raft_consensus.cc:504] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab5f1b922984bba8f33573db3cc9b3f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 39119 } }
I20260806 09:26:54.267985 10294 leader_election.cc:302] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ab5f1b922984bba8f33573db3cc9b3f; no voters:
I20260806 09:26:54.268920 10294 leader_election.cc:288] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:26:54.269174 10302 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:26:54.270083 10302 raft_consensus.cc:686] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [term 1 LEADER]: Becoming Leader. State: Replica: 9ab5f1b922984bba8f33573db3cc9b3f, State: Running, Role: LEADER
I20260806 09:26:54.270823 10302 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab5f1b922984bba8f33573db3cc9b3f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 39119 } }
I20260806 09:26:54.271842 10294 sys_catalog.cc:564] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:26:54.274550 10306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ab5f1b922984bba8f33573db3cc9b3f. Latest consensus state: current_term: 1 leader_uuid: "9ab5f1b922984bba8f33573db3cc9b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab5f1b922984bba8f33573db3cc9b3f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 39119 } } }
I20260806 09:26:54.275332 10306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [sys.catalog]: This master's current role is: LEADER
I20260806 09:26:54.275264 10303 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ab5f1b922984bba8f33573db3cc9b3f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ab5f1b922984bba8f33573db3cc9b3f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 39119 } } }
I20260806 09:26:54.275717 10303 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f [sys.catalog]: This master's current role is: LEADER
I20260806 09:26:54.278221 10312 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:26:54.285028 10312 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:26:54.297673 10312 catalog_manager.cc:1347] Generated new cluster ID: 06ac840298814a97b5fdbca1e2e3954f
I20260806 09:26:54.297870 10312 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:26:54.342752 10312 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:26:54.344954 10312 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:26:54.362270 10312 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f: Generated new TSK 0
I20260806 09:26:54.363722 10312 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:26:54.472956 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:39119
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:26:54.857643 10330 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:54.857962 10330 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:54.858114 10330 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:54.865628 10330 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:54.865767 10330 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:54.865955 10330 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:26:54.876544 10330 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:39119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:54.878037 10330 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:54.879578 10330 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:54.895118 10341 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:54.895308 10336 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:54.895164 10344 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:54.896677 10330 server_base.cc:1034] running on GCE node
I20260806 09:26:54.897722 10330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:54.899308 10330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:54.900652 10330 hybrid_clock.cc:648] HybridClock initialized: now 1786008414900580 us; error 51 us; skew 500 ppm
I20260806 09:26:54.901202 10330 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:54 dist-test-slave-h3ft krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008414, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:54.905541 10330 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:26:54.907639 10330 webserver.cc:460] Webserver started at http://127.16.86.129:39199/ using document root <none> and password file <none>
I20260806 09:26:54.908636 10330 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:54.908828 10330 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:54.909243 10330 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:54.913061 10330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "6012f86dc7024b83adb6d008c3c6f9ed"
format_stamp: "Formatted at 2026-08-06 09:26:54 on dist-test-slave-h3ft"
server_key: "34502d4efddd1dffde25326ab91d7c74"
server_key_iv: "3159434d022f491c359cd68eba7bd97c"
server_key_version: "encryptionkey@0"
I20260806 09:26:54.914026 10330 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "6012f86dc7024b83adb6d008c3c6f9ed"
format_stamp: "Formatted at 2026-08-06 09:26:54 on dist-test-slave-h3ft"
server_key: "34502d4efddd1dffde25326ab91d7c74"
server_key_iv: "3159434d022f491c359cd68eba7bd97c"
server_key_version: "encryptionkey@0"
I20260806 09:26:54.920944 10330 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.002s
I20260806 09:26:54.925663 10356 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:54.927654 10330 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:26:54.927917 10330 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "6012f86dc7024b83adb6d008c3c6f9ed"
format_stamp: "Formatted at 2026-08-06 09:26:54 on dist-test-slave-h3ft"
server_key: "34502d4efddd1dffde25326ab91d7c74"
server_key_iv: "3159434d022f491c359cd68eba7bd97c"
server_key_version: "encryptionkey@0"
I20260806 09:26:54.928203 10330 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:54.957386 10330 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:54.963331 10330 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:54.963743 10330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:54.965046 10330 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:54.966770 10330 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:54.966890 10330 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:54.967087 10330 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:54.967172 10330 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260806 09:26:55.011343 10330 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:45019
I20260806 09:26:55.011423 10508 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:45019 every 8 connection(s)
I20260806 09:26:55.013232 10330 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:26:55.021196 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 10330
I20260806 09:26:55.021481 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:26:55.022069 16730 external_mini_cluster.cc:1428] Setting key 1e7a0764d7f737d5f40f18409337565e
Aug 06 09:26:55 dist-test-slave-h3ft krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008414, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:26:55.039134 10512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:55.016767 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:45673 (local address 127.16.86.190:39119)
0806 09:26:55.017293 (+ 526us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:55.017298 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:55.018499 (+ 1201us) server_negotiation.cc:407] Connection header received
0806 09:26:55.020760 (+ 2261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:55.020766 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:55.020866 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:55.021001 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:55.023057 (+ 2056us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:55.024063 (+ 1006us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:55.026984 (+ 2921us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:55.027502 (+ 518us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:55.027638 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:55.032802 (+ 5164us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:55.032807 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:55.032811 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:55.032845 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:55.035772 (+ 2927us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:55.036594 (+ 822us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:55.036601 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:55.036612 (+ 11us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:55.036743 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:55.037219 (+ 476us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:55.037227 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:55.037230 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:55.037511 (+ 281us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:55.037684 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:55.038438 (+ 754us) server_negotiation.cc:299] Negotiation successful
0806 09:26:55.038686 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":403,"thread_start_us":291,"threads_started":1}
I20260806 09:26:55.040181 10513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:55.017352 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:39119 (local address 127.16.86.129:45673)
0806 09:26:55.018269 (+ 917us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:55.018328 (+ 59us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:55.020419 (+ 2091us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:55.021263 (+ 844us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:55.021271 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:55.022002 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:55.022740 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:55.022766 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:55.024347 (+ 1581us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:55.024353 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:55.026753 (+ 2400us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:55.026778 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:55.027807 (+ 1029us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:55.027817 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:55.028205 (+ 388us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:55.029160 (+ 955us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:26:55.029196 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:55.032537 (+ 3341us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:55.035937 (+ 3400us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:55.035947 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:55.035964 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:55.036464 (+ 500us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:55.036874 (+ 410us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:55.036879 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:55.036883 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:55.037085 (+ 202us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:55.037662 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:55.037670 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:55.038041 (+ 371us) client_negotiation.cc:769] Sending connection context
0806 09:26:55.038344 (+ 303us) client_negotiation.cc:240] Negotiation successful
0806 09:26:55.038924 (+ 580us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":618,"thread_start_us":350,"threads_started":1}
I20260806 09:26:55.042229 10509 heartbeater.cc:347] Connected to a master server at 127.16.86.190:39119
I20260806 09:26:55.042605 10509 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:55.043624 10509 heartbeater.cc:511] Master 127.16.86.190:39119 requested a full tablet report, sending...
I20260806 09:26:55.046391 10244 ts_manager.cc:194] Registered new tserver with Master: 6012f86dc7024b83adb6d008c3c6f9ed (127.16.86.129:45019)
I20260806 09:26:55.047972 10244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:45673
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:26:55.116398 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:39119
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:26:55.481843 10516 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:55.482220 10516 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:55.482391 10516 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:55.489919 10516 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:55.490120 10516 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:55.490324 10516 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:26:55.500590 10516 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:39119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:55.502064 10516 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:55.503481 10516 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:55.519753 10522 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:55.519630 10527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:55.519587 10524 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:55.524945 10516 server_base.cc:1034] running on GCE node
I20260806 09:26:55.525687 10516 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:55.526834 10516 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:55.528105 10516 hybrid_clock.cc:648] HybridClock initialized: now 1786008415528038 us; error 69 us; skew 500 ppm
I20260806 09:26:55.528474 10516 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:55 dist-test-slave-h3ft krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008415, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:55.532218 10516 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:26:55.534220 10516 webserver.cc:460] Webserver started at http://127.16.86.130:44893/ using document root <none> and password file <none>
I20260806 09:26:55.535181 10516 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:55.535346 10516 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:55.535708 10516 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:55.538798 10516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "a6e71804677e4b828b55386ed6d9f5f4"
format_stamp: "Formatted at 2026-08-06 09:26:55 on dist-test-slave-h3ft"
server_key: "a41a5fb929999de3ba296b50dcaeb86f"
server_key_iv: "ab1b6fb429f08ac7e2d6cfdf7ea80e69"
server_key_version: "encryptionkey@0"
I20260806 09:26:55.539763 10516 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "a6e71804677e4b828b55386ed6d9f5f4"
format_stamp: "Formatted at 2026-08-06 09:26:55 on dist-test-slave-h3ft"
server_key: "a41a5fb929999de3ba296b50dcaeb86f"
server_key_iv: "ab1b6fb429f08ac7e2d6cfdf7ea80e69"
server_key_version: "encryptionkey@0"
I20260806 09:26:55.546129 10516 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:26:55.550508 10537 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:55.552327 10516 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:26:55.552500 10516 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "a6e71804677e4b828b55386ed6d9f5f4"
format_stamp: "Formatted at 2026-08-06 09:26:55 on dist-test-slave-h3ft"
server_key: "a41a5fb929999de3ba296b50dcaeb86f"
server_key_iv: "ab1b6fb429f08ac7e2d6cfdf7ea80e69"
server_key_version: "encryptionkey@0"
I20260806 09:26:55.552731 10516 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:55.574757 10516 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:55.579769 10516 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:55.580042 10516 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:55.581139 10516 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:55.582635 10516 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:55.582727 10516 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:55.582823 10516 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:55.582876 10516 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:55.625360 10516 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:33239
I20260806 09:26:55.625396 10685 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:33239 every 8 connection(s)
I20260806 09:26:55.626811 10516 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:26:55.637055 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 10516
I20260806 09:26:55.637296 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:26:55.637790 16730 external_mini_cluster.cc:1428] Setting key 8e30759303b3b7c99003417af6849245
Aug 06 09:26:55 dist-test-slave-h3ft krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008415, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:26:55.657647 10690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:55.630405 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:44119 (local address 127.16.86.190:39119)
0806 09:26:55.631204 (+ 799us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:55.631213 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:55.632053 (+ 840us) server_negotiation.cc:407] Connection header received
0806 09:26:55.634783 (+ 2730us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:55.634789 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:55.634929 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:55.635116 (+ 187us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:55.637210 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:55.638279 (+ 1069us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:55.640332 (+ 2053us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:55.640867 (+ 535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:55.641046 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:55.646771 (+ 5725us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:55.646783 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:55.646791 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:55.646865 (+ 74us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:55.651647 (+ 4782us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:55.653479 (+ 1832us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:55.653488 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:55.653494 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:55.653698 (+ 204us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:55.654507 (+ 809us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:55.654516 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:55.654523 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:55.655083 (+ 560us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:55.655305 (+ 222us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:55.656694 (+ 1389us) server_negotiation.cc:299] Negotiation successful
0806 09:26:55.657057 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":550,"thread_start_us":363,"threads_started":1}
I20260806 09:26:55.659363 10691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:55.630732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:39119 (local address 127.16.86.130:44119)
0806 09:26:55.631793 (+ 1061us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:55.631850 (+ 57us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:55.634365 (+ 2515us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:55.635483 (+ 1118us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:55.635494 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:55.636205 (+ 711us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:55.636984 (+ 779us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:55.637008 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:55.638460 (+ 1452us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:55.638472 (+ 12us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:55.640145 (+ 1673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:55.640161 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:55.641209 (+ 1048us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:55.641219 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:55.641530 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:55.642482 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:26:55.642521 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:55.646279 (+ 3758us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:55.652048 (+ 5769us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:55.652066 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:55.652096 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:55.653218 (+ 1122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:55.653978 (+ 760us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:55.653985 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:55.653991 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:55.654324 (+ 333us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:55.655289 (+ 965us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:55.655303 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:55.655960 (+ 657us) client_negotiation.cc:769] Sending connection context
0806 09:26:55.656426 (+ 466us) client_negotiation.cc:240] Negotiation successful
0806 09:26:55.657334 (+ 908us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":711,"thread_start_us":437,"threads_started":1}
I20260806 09:26:55.661659 10687 heartbeater.cc:347] Connected to a master server at 127.16.86.190:39119
I20260806 09:26:55.662118 10687 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:55.663272 10687 heartbeater.cc:511] Master 127.16.86.190:39119 requested a full tablet report, sending...
I20260806 09:26:55.665800 10244 ts_manager.cc:194] Registered new tserver with Master: a6e71804677e4b828b55386ed6d9f5f4 (127.16.86.130:33239)
I20260806 09:26:55.666919 10244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:44119
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:26:55.724503 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:39119
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:46595
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:26:56.051216 10509 heartbeater.cc:503] Master 127.16.86.190:39119 was elected leader, sending a full tablet report...
W20260806 09:26:56.120450 10696 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:56.120848 10696 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:56.120954 10696 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:56.128803 10696 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:56.128943 10696 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:56.129098 10696 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:26:56.139564 10696 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46595
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:39119
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:56.140985 10696 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:56.142357 10696 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:56.157943 10705 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:56.158031 10703 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:56.158699 10702 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:56.159165 10696 server_base.cc:1034] running on GCE node
I20260806 09:26:56.160027 10696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:56.161306 10696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:56.162541 10696 hybrid_clock.cc:648] HybridClock initialized: now 1786008416162461 us; error 75 us; skew 500 ppm
I20260806 09:26:56.162889 10696 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:26:56 dist-test-slave-h3ft krb5kdc[10185](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008416, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:26:56.167136 10696 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:26:56.169006 10696 webserver.cc:460] Webserver started at http://127.16.86.131:37443/ using document root <none> and password file <none>
I20260806 09:26:56.169818 10696 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:56.169957 10696 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:56.170333 10696 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:56.173833 10696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff"
format_stamp: "Formatted at 2026-08-06 09:26:56 on dist-test-slave-h3ft"
server_key: "ece1eec304b8188a8db3d9f5528563ac"
server_key_iv: "5931900e82bf1369f7ddf20c72ac9a47"
server_key_version: "encryptionkey@0"
I20260806 09:26:56.174659 10696 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff"
format_stamp: "Formatted at 2026-08-06 09:26:56 on dist-test-slave-h3ft"
server_key: "ece1eec304b8188a8db3d9f5528563ac"
server_key_iv: "5931900e82bf1369f7ddf20c72ac9a47"
server_key_version: "encryptionkey@0"
I20260806 09:26:56.180941 10696 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.002s
I20260806 09:26:56.185250 10714 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:56.187171 10696 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:26:56.187384 10696 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff"
format_stamp: "Formatted at 2026-08-06 09:26:56 on dist-test-slave-h3ft"
server_key: "ece1eec304b8188a8db3d9f5528563ac"
server_key_iv: "5931900e82bf1369f7ddf20c72ac9a47"
server_key_version: "encryptionkey@0"
I20260806 09:26:56.187662 10696 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:56.219899 10696 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:56.224565 10696 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:56.224888 10696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:56.226053 10696 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:56.227629 10696 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:56.227723 10696 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:56.227960 10696 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:56.228060 10696 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:56.270772 10696 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:43857
I20260806 09:26:56.270905 10875 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:43857 every 8 connection(s)
I20260806 09:26:56.272446 10696 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:26:56.273597 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 10696
I20260806 09:26:56.273844 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:26:56.274488 16730 external_mini_cluster.cc:1428] Setting key c6cbc4e92e9232a0a799f3df78af4986
Aug 06 09:26:56 dist-test-slave-h3ft krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008416, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:26:56.299165 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.276224 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:50877 (local address 127.16.86.190:39119)
0806 09:26:56.276949 (+ 725us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:56.276958 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:56.278362 (+ 1404us) server_negotiation.cc:407] Connection header received
0806 09:26:56.280868 (+ 2506us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:56.280872 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:56.280954 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:56.281064 (+ 110us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:56.283233 (+ 2169us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.284075 (+ 842us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.285955 (+ 1880us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.286380 (+ 425us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.286485 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.291552 (+ 5067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:56.291621 (+ 69us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:56.291627 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:56.291698 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:56.295246 (+ 3548us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:56.296208 (+ 962us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:56.296218 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:56.296221 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:56.296371 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:56.296938 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:56.296948 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:56.296950 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:56.297284 (+ 334us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:56.297480 (+ 196us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:56.298353 (+ 873us) server_negotiation.cc:299] Negotiation successful
0806 09:26:56.298632 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"thread_start_us":276,"threads_started":1}
I20260806 09:26:56.300432 10879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.276921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:39119 (local address 127.16.86.131:50877)
0806 09:26:56.278099 (+ 1178us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:56.278217 (+ 118us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:56.280536 (+ 2319us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:56.281411 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:56.281422 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:56.282128 (+ 706us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:56.282951 (+ 823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:56.282976 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.284287 (+ 1311us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.284292 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:56.285776 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:56.285792 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.286672 (+ 880us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.286679 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:56.287090 (+ 411us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.288042 (+ 952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:26:56.288073 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:56.291258 (+ 3185us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:56.295443 (+ 4185us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:56.295455 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:56.295480 (+ 25us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:56.296037 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:56.296527 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:26:56.296532 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:26:56.296536 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:26:56.296797 (+ 261us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:26:56.297436 (+ 639us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:56.297446 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:56.297928 (+ 482us) client_negotiation.cc:769] Sending connection context
0806 09:26:56.298259 (+ 331us) client_negotiation.cc:240] Negotiation successful
0806 09:26:56.298976 (+ 717us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":707,"thread_start_us":481,"threads_started":1}
I20260806 09:26:56.302145 10876 heartbeater.cc:347] Connected to a master server at 127.16.86.190:39119
I20260806 09:26:56.302537 10876 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:56.303613 10876 heartbeater.cc:511] Master 127.16.86.190:39119 requested a full tablet report, sending...
I20260806 09:26:56.305576 10244 ts_manager.cc:194] Registered new tserver with Master: b76e0cf1ac8441fcbfee9fd76a2fc9ff (127.16.86.131:43857)
I20260806 09:26:56.306525 10244 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:50877
I20260806 09:26:56.318106 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:26:56.340014 10878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.323528 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35028 (local address 127.16.86.190:39119)
0806 09:26:56.323876 (+ 348us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:56.323887 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:56.324404 (+ 517us) server_negotiation.cc:407] Connection header received
0806 09:26:56.324660 (+ 256us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:56.324664 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:56.324777 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:56.324904 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:56.326580 (+ 1676us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.327490 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.328989 (+ 1499us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.329443 (+ 454us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.329623 (+ 180us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.333197 (+ 3574us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:56.333203 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:56.333208 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:56.333248 (+ 40us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:56.336663 (+ 3415us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:56.337573 (+ 910us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:56.337579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:56.337583 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:56.337728 (+ 145us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:56.338250 (+ 522us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:56.338261 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:56.338263 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:56.338629 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:56.338955 (+ 326us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:56.339433 (+ 478us) server_negotiation.cc:299] Negotiation successful
0806 09:26:56.339677 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260806 09:26:56.345774 10244 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35028:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:26:56.354803 10244 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:26:56.390137 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.373889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43386 (local address 127.16.86.130:33239)
0806 09:26:56.375935 (+ 2046us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:56.375952 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:56.379790 (+ 3838us) server_negotiation.cc:407] Connection header received
0806 09:26:56.380187 (+ 397us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:56.380198 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:56.380565 (+ 367us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:56.380785 (+ 220us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:56.381689 (+ 904us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.383486 (+ 1797us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.386647 (+ 3161us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.387815 (+ 1168us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.388000 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.388271 (+ 271us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:56.389458 (+ 1187us) server_negotiation.cc:299] Negotiation successful
0806 09:26:56.389679 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1740,"thread_start_us":2491,"threads_started":1}
I20260806 09:26:56.391376 10897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.373599 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:33239 (local address 127.0.0.1:43386)
0806 09:26:56.379636 (+ 6037us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:56.379658 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:56.379944 (+ 286us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:56.380990 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:56.380998 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:56.381031 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:56.381484 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:56.381495 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.383687 (+ 2192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.383695 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:56.386445 (+ 2750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:56.386462 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.388077 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.388085 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:56.389103 (+ 1018us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.389176 (+ 73us) client_negotiation.cc:769] Sending connection context
0806 09:26:56.390507 (+ 1331us) client_negotiation.cc:240] Negotiation successful
0806 09:26:56.390774 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5880,"mutex_wait_us":700,"thread_start_us":5792,"threads_started":1}
I20260806 09:26:56.392287 10894 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.373018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:43857 (local address 127.0.0.1:40244)
0806 09:26:56.374053 (+ 1035us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:56.374205 (+ 152us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:56.375057 (+ 852us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:56.383118 (+ 8061us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:56.383123 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:56.383157 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:56.383582 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:56.383595 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.386196 (+ 2601us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.386203 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:56.388728 (+ 2525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:56.388745 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.390876 (+ 2131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.390882 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:56.391236 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.391278 (+ 42us) client_negotiation.cc:769] Sending connection context
0806 09:26:56.391394 (+ 116us) client_negotiation.cc:240] Negotiation successful
0806 09:26:56.391801 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"mutex_wait_us":68,"thread_start_us":1154,"threads_started":1}
I20260806 09:26:56.392565 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.376127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33150 (local address 127.16.86.129:45019)
0806 09:26:56.377139 (+ 1012us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:56.377152 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:56.382922 (+ 5770us) server_negotiation.cc:407] Connection header received
0806 09:26:56.383260 (+ 338us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:56.383271 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:56.383597 (+ 326us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:56.383783 (+ 186us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:56.384860 (+ 1077us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.386012 (+ 1152us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.389667 (+ 3655us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.391061 (+ 1394us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.391213 (+ 152us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.391569 (+ 356us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:56.391929 (+ 360us) server_negotiation.cc:299] Negotiation successful
0806 09:26:56.392124 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":732,"thread_start_us":540,"threads_started":1}
I20260806 09:26:56.393325 10899 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.375335 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:45019 (local address 127.0.0.1:33150)
0806 09:26:56.382756 (+ 7421us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:56.382774 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:56.382988 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:56.384063 (+ 1075us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:56.384067 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:56.384091 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:56.384627 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:56.384642 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.386214 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.386218 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:56.389471 (+ 3253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:56.389488 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.391287 (+ 1799us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.391297 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:56.391635 (+ 338us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.391688 (+ 53us) client_negotiation.cc:769] Sending connection context
0806 09:26:56.392909 (+ 1221us) client_negotiation.cc:240] Negotiation successful
0806 09:26:56.393033 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7260,"mutex_wait_us":754,"thread_start_us":7097,"threads_started":1}
I20260806 09:26:56.393437 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.373221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40244 (local address 127.16.86.131:43857)
0806 09:26:56.382364 (+ 9143us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:56.382375 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:56.382427 (+ 52us) server_negotiation.cc:407] Connection header received
0806 09:26:56.382544 (+ 117us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:56.382553 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:56.382817 (+ 264us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:56.383124 (+ 307us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:56.384306 (+ 1182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.386024 (+ 1718us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.388930 (+ 2906us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.390627 (+ 1697us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.390770 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.391054 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:56.392803 (+ 1749us) server_negotiation.cc:299] Negotiation successful
0806 09:26:56.393058 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8919,"thread_start_us":8731,"threads_started":1}
I20260806 09:26:56.395968 10596 tablet_service.cc:1467] Processing CreateTablet for tablet 59d09e4d78994b5ba5d8b57ecc6432ad (DEFAULT_TABLE table=test-table [id=0f8e341b1344478ea2b7aa25cd628d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:56.397684 10596 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d09e4d78994b5ba5d8b57ecc6432ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:56.397836 10417 tablet_service.cc:1467] Processing CreateTablet for tablet 59d09e4d78994b5ba5d8b57ecc6432ad (DEFAULT_TABLE table=test-table [id=0f8e341b1344478ea2b7aa25cd628d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:56.399477 10417 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d09e4d78994b5ba5d8b57ecc6432ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:56.399412 10784 tablet_service.cc:1467] Processing CreateTablet for tablet 59d09e4d78994b5ba5d8b57ecc6432ad (DEFAULT_TABLE table=test-table [id=0f8e341b1344478ea2b7aa25cd628d6f]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:56.401536 10784 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 59d09e4d78994b5ba5d8b57ecc6432ad. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:56.413583 10901 tablet_bootstrap.cc:492] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed: Bootstrap starting.
I20260806 09:26:56.417690 10901 tablet_bootstrap.cc:654] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:56.419466 10901 log.cc:824] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:56.421837 10902 tablet_bootstrap.cc:492] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4: Bootstrap starting.
I20260806 09:26:56.426959 10902 tablet_bootstrap.cc:654] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:56.428844 10902 log.cc:824] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:56.430356 10903 tablet_bootstrap.cc:492] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff: Bootstrap starting.
I20260806 09:26:56.434363 10903 tablet_bootstrap.cc:654] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:56.435796 10903 log.cc:824] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:56.437322 10902 tablet_bootstrap.cc:492] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4: No bootstrap required, opened a new log
I20260806 09:26:56.437676 10902 ts_tablet_manager.cc:1397] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4: Time spent bootstrapping tablet: real 0.018s user 0.004s sys 0.004s
I20260806 09:26:56.437767 10901 tablet_bootstrap.cc:492] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed: No bootstrap required, opened a new log
I20260806 09:26:56.438153 10901 ts_tablet_manager.cc:1397] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed: Time spent bootstrapping tablet: real 0.025s user 0.008s sys 0.013s
I20260806 09:26:56.439625 10903 tablet_bootstrap.cc:492] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff: No bootstrap required, opened a new log
I20260806 09:26:56.439961 10903 ts_tablet_manager.cc:1397] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20260806 09:26:56.445215 10902 raft_consensus.cc:348] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43857 } } peers { permanent_uuid: "a6e71804677e4b828b55386ed6d9f5f4" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33239 } } peers { permanent_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45019 } }
I20260806 09:26:56.445662 10902 raft_consensus.cc:374] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:56.445760 10902 raft_consensus.cc:729] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a6e71804677e4b828b55386ed6d9f5f4, State: Initialized, Role: FOLLOWER
I20260806 09:26:56.446488 10902 consensus_queue.cc:260] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43857 } } peers { permanent_uuid: "a6e71804677e4b828b55386ed6d9f5f4" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33239 } } peers { permanent_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45019 } }
I20260806 09:26:56.447834 10687 heartbeater.cc:503] Master 127.16.86.190:39119 was elected leader, sending a full tablet report...
I20260806 09:26:56.448961 10902 ts_tablet_manager.cc:1428] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4: Time spent starting tablet: real 0.011s user 0.009s sys 0.004s
I20260806 09:26:56.450459 10901 raft_consensus.cc:348] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43857 } } peers { permanent_uuid: "a6e71804677e4b828b55386ed6d9f5f4" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33239 } } peers { permanent_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45019 } }
I20260806 09:26:56.451107 10901 raft_consensus.cc:374] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:56.451236 10901 raft_consensus.cc:729] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6012f86dc7024b83adb6d008c3c6f9ed, State: Initialized, Role: FOLLOWER
I20260806 09:26:56.452095 10901 consensus_queue.cc:260] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43857 } } peers { permanent_uuid: "a6e71804677e4b828b55386ed6d9f5f4" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33239 } } peers { permanent_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45019 } }
I20260806 09:26:56.453717 10903 raft_consensus.cc:348] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43857 } } peers { permanent_uuid: "a6e71804677e4b828b55386ed6d9f5f4" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33239 } } peers { permanent_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45019 } }
I20260806 09:26:56.454296 10903 raft_consensus.cc:374] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:56.454425 10903 raft_consensus.cc:729] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b76e0cf1ac8441fcbfee9fd76a2fc9ff, State: Initialized, Role: FOLLOWER
I20260806 09:26:56.454618 10901 ts_tablet_manager.cc:1428] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed: Time spent starting tablet: real 0.016s user 0.017s sys 0.000s
I20260806 09:26:56.455399 10903 consensus_queue.cc:260] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43857 } } peers { permanent_uuid: "a6e71804677e4b828b55386ed6d9f5f4" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33239 } } peers { permanent_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45019 } }
I20260806 09:26:56.456797 10876 heartbeater.cc:503] Master 127.16.86.190:39119 was elected leader, sending a full tablet report...
I20260806 09:26:56.457469 10903 ts_tablet_manager.cc:1428] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff: Time spent starting tablet: real 0.017s user 0.014s sys 0.000s
W20260806 09:26:56.516786 10510 tablet.cc:2369] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:26:56.524710 10877 tablet.cc:2369] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:26:56.548772 10913 raft_consensus.cc:482] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:26:56.549129 10913 raft_consensus.cc:504] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43857 } } peers { permanent_uuid: "a6e71804677e4b828b55386ed6d9f5f4" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33239 } } peers { permanent_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45019 } }
I20260806 09:26:56.551203 10913 leader_election.cc:288] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b76e0cf1ac8441fcbfee9fd76a2fc9ff (127.16.86.131:43857), a6e71804677e4b828b55386ed6d9f5f4 (127.16.86.130:33239)
I20260806 09:26:56.559484 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.552437 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:52475 (local address 127.16.86.130:33239)
0806 09:26:56.552757 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:56.552763 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:56.553130 (+ 367us) server_negotiation.cc:407] Connection header received
0806 09:26:56.553384 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:56.553388 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:56.553452 (+ 64us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:56.553563 (+ 111us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:56.554349 (+ 786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.555182 (+ 833us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.557456 (+ 2274us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.558409 (+ 953us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.558633 (+ 224us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.558718 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:56.559056 (+ 338us) server_negotiation.cc:299] Negotiation successful
0806 09:26:56.559191 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":122}
I20260806 09:26:56.559693 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.551668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:48407 (local address 127.16.86.131:43857)
0806 09:26:56.551958 (+ 290us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:56.551966 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:56.552996 (+ 1030us) server_negotiation.cc:407] Connection header received
0806 09:26:56.553288 (+ 292us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:56.553292 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:56.553368 (+ 76us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:56.553481 (+ 113us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:56.554194 (+ 713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.555097 (+ 903us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.557475 (+ 2378us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.558498 (+ 1023us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.558734 (+ 236us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.558825 (+ 91us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:56.559200 (+ 375us) server_negotiation.cc:299] Negotiation successful
0806 09:26:56.559347 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":96}
I20260806 09:26:56.559715 10917 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.551769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:33239 (local address 127.16.86.129:52475)
0806 09:26:56.552990 (+ 1221us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:56.553011 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:56.553213 (+ 202us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:56.553584 (+ 371us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:56.553589 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:56.553624 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:56.554178 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:56.554190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.555332 (+ 1142us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.555338 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:56.557287 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:56.557301 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.558648 (+ 1347us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.558654 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:56.558835 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.558865 (+ 30us) client_negotiation.cc:769] Sending connection context
0806 09:26:56.558986 (+ 121us) client_negotiation.cc:240] Negotiation successful
0806 09:26:56.559156 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1056,"mutex_wait_us":7,"thread_start_us":523,"threads_started":1}
I20260806 09:26:56.560322 10918 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.551686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:43857 (local address 127.16.86.129:48407)
0806 09:26:56.552822 (+ 1136us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:56.552854 (+ 32us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:56.553107 (+ 253us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:56.553500 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:56.553505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:56.553548 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:56.554022 (+ 474us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:56.554037 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.555237 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.555245 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:56.557289 (+ 2044us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:56.557303 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.558762 (+ 1459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.558768 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:56.558959 (+ 191us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.558984 (+ 25us) client_negotiation.cc:769] Sending connection context
0806 09:26:56.559168 (+ 184us) client_negotiation.cc:240] Negotiation successful
0806 09:26:56.559357 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":972,"mutex_wait_us":26,"thread_start_us":1092,"threads_started":1}
I20260806 09:26:56.561007 10622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d09e4d78994b5ba5d8b57ecc6432ad" candidate_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e71804677e4b828b55386ed6d9f5f4" is_pre_election: true
I20260806 09:26:56.561511 10815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d09e4d78994b5ba5d8b57ecc6432ad" candidate_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff" is_pre_election: true
I20260806 09:26:56.561721 10622 raft_consensus.cc:2454] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6012f86dc7024b83adb6d008c3c6f9ed in term 0.
I20260806 09:26:56.562115 10815 raft_consensus.cc:2454] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 6012f86dc7024b83adb6d008c3c6f9ed in term 0.
I20260806 09:26:56.562760 10357 leader_election.cc:302] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6012f86dc7024b83adb6d008c3c6f9ed, a6e71804677e4b828b55386ed6d9f5f4; no voters:
I20260806 09:26:56.563391 10913 raft_consensus.cc:2781] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:26:56.563603 10913 raft_consensus.cc:482] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:26:56.563777 10913 raft_consensus.cc:3037] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:56.566201 10913 raft_consensus.cc:504] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43857 } } peers { permanent_uuid: "a6e71804677e4b828b55386ed6d9f5f4" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33239 } } peers { permanent_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45019 } }
I20260806 09:26:56.566948 10913 leader_election.cc:288] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [CANDIDATE]: Term 1 election: Requested vote from peers b76e0cf1ac8441fcbfee9fd76a2fc9ff (127.16.86.131:43857), a6e71804677e4b828b55386ed6d9f5f4 (127.16.86.130:33239)
I20260806 09:26:56.567584 10815 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d09e4d78994b5ba5d8b57ecc6432ad" candidate_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff"
I20260806 09:26:56.567808 10815 raft_consensus.cc:3037] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:56.567785 10622 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "59d09e4d78994b5ba5d8b57ecc6432ad" candidate_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a6e71804677e4b828b55386ed6d9f5f4"
I20260806 09:26:56.568099 10622 raft_consensus.cc:3037] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:56.570204 10815 raft_consensus.cc:2454] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6012f86dc7024b83adb6d008c3c6f9ed in term 1.
I20260806 09:26:56.570706 10361 leader_election.cc:302] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6012f86dc7024b83adb6d008c3c6f9ed, b76e0cf1ac8441fcbfee9fd76a2fc9ff; no voters:
I20260806 09:26:56.571173 10913 raft_consensus.cc:2781] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:26:56.571391 10622 raft_consensus.cc:2454] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 6012f86dc7024b83adb6d008c3c6f9ed in term 1.
I20260806 09:26:56.571930 10913 raft_consensus.cc:686] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 1 LEADER]: Becoming Leader. State: Replica: 6012f86dc7024b83adb6d008c3c6f9ed, State: Running, Role: LEADER
I20260806 09:26:56.572723 10913 consensus_queue.cc:237] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43857 } } peers { permanent_uuid: "a6e71804677e4b828b55386ed6d9f5f4" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33239 } } peers { permanent_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45019 } }
I20260806 09:26:56.578658 10243 catalog_manager.cc:5495] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed reported cstate change: term changed from 0 to 1, leader changed from <none> to 6012f86dc7024b83adb6d008c3c6f9ed (127.16.86.129). New cstate: current_term: 1 leader_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43857 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a6e71804677e4b828b55386ed6d9f5f4" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33239 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6012f86dc7024b83adb6d008c3c6f9ed" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 45019 } health_report { overall_health: HEALTHY } } }
W20260806 09:26:56.629835 10688 tablet.cc:2369] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:26:56.642827 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.635909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33152 (local address 127.16.86.129:45019)
0806 09:26:56.636137 (+ 228us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:56.636143 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:56.636301 (+ 158us) server_negotiation.cc:407] Connection header received
0806 09:26:56.636544 (+ 243us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:56.636548 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:56.636634 (+ 86us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:56.636773 (+ 139us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:26:56.637291 (+ 518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.638013 (+ 722us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.640258 (+ 2245us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.640669 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.640777 (+ 108us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.641509 (+ 732us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:26:56.642057 (+ 548us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:26:56.642223 (+ 166us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:56.642437 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 09:26:56.642482 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20260806 09:26:56.653199 10622 raft_consensus.cc:1261] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4 [term 1 FOLLOWER]: Refusing update from remote peer 6012f86dc7024b83adb6d008c3c6f9ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:26:56.653239 10815 raft_consensus.cc:1261] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff [term 1 FOLLOWER]: Refusing update from remote peer 6012f86dc7024b83adb6d008c3c6f9ed: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:26:56.654546 10913 consensus_queue.cc:1035] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "b76e0cf1ac8441fcbfee9fd76a2fc9ff" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43857 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:26:56.655126 10919 consensus_queue.cc:1035] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [LEADER]: Connected to new peer: Peer: permanent_uuid: "a6e71804677e4b828b55386ed6d9f5f4" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 33239 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:26:56.670907 10923 mvcc.cc:205] Tried to move back new op lower bound from 7315490474597376000 to 7315490474287443968. Current Snapshot: MvccSnapshot[applied={T|T < 7315490474597376000}]
I20260806 09:26:56.687829 10243 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:26:56.688305 10243 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:35028:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:26:56.693534 10243 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 9ab5f1b922984bba8f33573db3cc9b3f: Sending DeleteTablet for 3 replicas of tablet 59d09e4d78994b5ba5d8b57ecc6432ad
I20260806 09:26:56.695372 10417 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d09e4d78994b5ba5d8b57ecc6432ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:26:56 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:33150
I20260806 09:26:56.695423 10596 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d09e4d78994b5ba5d8b57ecc6432ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:26:56 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:43386
I20260806 09:26:56.696625 10942 tablet_replica.cc:330] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed: stopping tablet replica
I20260806 09:26:56.697291 10942 raft_consensus.cc:2229] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:26:56.697703 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 10330
I20260806 09:26:56.698470 10942 raft_consensus.cc:2258] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:26:56.699558 10941 tablet_replica.cc:330] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4: stopping tablet replica
I20260806 09:26:56.700126 10941 raft_consensus.cc:2229] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:26:56.700726 10941 raft_consensus.cc:2258] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:26:56.700618 10784 tablet_service.cc:1514] Processing DeleteTablet for tablet 59d09e4d78994b5ba5d8b57ecc6432ad with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:26:56 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:40244
I20260806 09:26:56.703186 10941 ts_tablet_manager.cc:1905] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:26:56.704035 10944 tablet_replica.cc:330] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff: stopping tablet replica
I20260806 09:26:56.704605 10944 raft_consensus.cc:2229] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:26:56.705309 10944 raft_consensus.cc:2258] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:26:56.708003 10944 ts_tablet_manager.cc:1905] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:26:56.708925 10942 ts_tablet_manager.cc:1905] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:26:56.711522 10941 ts_tablet_manager.cc:1918] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:26:56.711745 10941 log.cc:1196] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/59d09e4d78994b5ba5d8b57ecc6432ad
I20260806 09:26:56.712407 10941 ts_tablet_manager.cc:1939] T 59d09e4d78994b5ba5d8b57ecc6432ad P a6e71804677e4b828b55386ed6d9f5f4: Deleting consensus metadata
I20260806 09:26:56.715102 10223 catalog_manager.cc:4841] TS a6e71804677e4b828b55386ed6d9f5f4 (127.16.86.130:33239): tablet 59d09e4d78994b5ba5d8b57ecc6432ad (table test-table [id=0f8e341b1344478ea2b7aa25cd628d6f]) successfully deleted
I20260806 09:26:56.716246 10944 ts_tablet_manager.cc:1918] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:26:56.716487 10944 log.cc:1196] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/59d09e4d78994b5ba5d8b57ecc6432ad
I20260806 09:26:56.716989 10944 ts_tablet_manager.cc:1939] T 59d09e4d78994b5ba5d8b57ecc6432ad P b76e0cf1ac8441fcbfee9fd76a2fc9ff: Deleting consensus metadata
I20260806 09:26:56.719141 10226 catalog_manager.cc:4841] TS b76e0cf1ac8441fcbfee9fd76a2fc9ff (127.16.86.131:43857): tablet 59d09e4d78994b5ba5d8b57ecc6432ad (table test-table [id=0f8e341b1344478ea2b7aa25cd628d6f]) successfully deleted
I20260806 09:26:56.722332 10942 ts_tablet_manager.cc:1918] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:26:56.722601 10942 log.cc:1196] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/59d09e4d78994b5ba5d8b57ecc6432ad
I20260806 09:26:56.723294 10942 ts_tablet_manager.cc:1939] T 59d09e4d78994b5ba5d8b57ecc6432ad P 6012f86dc7024b83adb6d008c3c6f9ed: Deleting consensus metadata
I20260806 09:26:56.725559 10226 catalog_manager.cc:4841] TS 6012f86dc7024b83adb6d008c3c6f9ed (127.16.86.129:45019): tablet 59d09e4d78994b5ba5d8b57ecc6432ad (table test-table [id=0f8e341b1344478ea2b7aa25cd628d6f]) successfully deleted
Aug 06 09:26:56 dist-test-slave-h3ft krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:26:56.739918 10900 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.699998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33154 (local address 127.16.86.129:45019)
0806 09:26:56.700227 (+ 229us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:56.700237 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:56.720120 (+ 19883us) server_negotiation.cc:407] Connection header received
0806 09:26:56.720457 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:56.720463 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:56.720604 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:56.720723 (+ 119us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:56.722147 (+ 1424us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.723347 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.724798 (+ 1451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.725389 (+ 591us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.725538 (+ 149us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.731513 (+ 5975us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:56.731533 (+ 20us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:56.731554 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:56.731624 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:56.736283 (+ 4659us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:56.737282 (+ 999us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:56.737291 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:56.737294 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:56.737487 (+ 193us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:56.737930 (+ 443us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:56.737938 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:56.737941 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:56.738408 (+ 467us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:56.738785 (+ 377us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:56.739194 (+ 409us) server_negotiation.cc:299] Negotiation successful
0806 09:26:56.739480 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:26:56.740866 10504 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:56.854563 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 10330
I20260806 09:26:56.886397 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 10516
Aug 06 09:26:56 dist-test-slave-h3ft krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:26:56.906330 10898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:56.887513 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43388 (local address 127.16.86.130:33239)
0806 09:26:56.887766 (+ 253us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:56.887774 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:56.887827 (+ 53us) server_negotiation.cc:407] Connection header received
0806 09:26:56.888084 (+ 257us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:56.888089 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:56.888177 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:56.888308 (+ 131us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:56.889798 (+ 1490us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.890574 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.892408 (+ 1834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:56.893037 (+ 629us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:56.893208 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:56.898447 (+ 5239us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:56.898471 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:56.898496 (+ 25us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:56.898556 (+ 60us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:56.902555 (+ 3999us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:56.903595 (+ 1040us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:56.903605 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:56.903608 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:56.903815 (+ 207us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:56.904375 (+ 560us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:56.904379 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:56.904381 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:56.904987 (+ 606us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:56.905292 (+ 305us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:56.905598 (+ 306us) server_negotiation.cc:299] Negotiation successful
0806 09:26:56.905865 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":97}
I20260806 09:26:56.907407 10679 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:57.019381 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 10516
I20260806 09:26:57.048940 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 10696
Aug 06 09:26:57 dist-test-slave-h3ft krb5kdc[10185](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008413, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:26:57.068446 10895 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:57.049998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40260 (local address 127.16.86.131:43857)
0806 09:26:57.050219 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:57.050224 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:57.050250 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:26:57.050421 (+ 171us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:57.050426 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:57.050506 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:57.050645 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:57.052608 (+ 1963us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:57.053386 (+ 778us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:57.055145 (+ 1759us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:57.055586 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:57.055766 (+ 180us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:57.061601 (+ 5835us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:57.061644 (+ 43us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:57.061662 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:26:57.061711 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:57.064977 (+ 3266us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:57.065848 (+ 871us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:57.065856 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:57.065858 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:57.066025 (+ 167us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:26:57.066535 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:26:57.066540 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:26:57.066543 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:26:57.066983 (+ 440us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:57.067472 (+ 489us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:57.067898 (+ 426us) server_negotiation.cc:299] Negotiation successful
0806 09:26:57.068125 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:26:57.069368 10870 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:57.173802 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 10696
I20260806 09:26:57.197685 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 10199
I20260806 09:26:57.199184 10288 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:26:57.316248 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 10199
2026-08-06T09:26:57Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5396 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2026-08-06T09:26:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:26:57Z Disabled control of system clock
I20260806 09:26:57.433159 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:43329
--webserver_interface=127.16.86.190
--webserver_port=0
--builtin_ntp_servers=127.16.86.148:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:43329
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:26:57.875988 10956 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:57.876504 10956 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:57.876775 10956 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:57.890017 10956 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:26:57.890202 10956 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:57.890277 10956 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:57.890373 10956 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:26:57.890467 10956 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:26:57.901209 10956 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:43329
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:43329
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:57.902700 10956 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:57.904690 10956 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:57.919924 10967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:57.920253 10964 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:57.919924 10962 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:57.920994 10956 server_base.cc:1034] running on GCE node
I20260806 09:26:57.921885 10956 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:57.923568 10956 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:57.924795 10956 hybrid_clock.cc:648] HybridClock initialized: now 1786008417924709 us; error 76 us; skew 500 ppm
I20260806 09:26:57.925202 10956 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:26:57.928503 10956 webserver.cc:460] Webserver started at http://127.16.86.190:36039/ using document root <none> and password file <none>
I20260806 09:26:57.929361 10956 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:57.929473 10956 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:57.929916 10956 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:57.933491 10956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "c49c706faf5540909464fb53a1064c6f"
format_stamp: "Formatted at 2026-08-06 09:26:57 on dist-test-slave-h3ft"
server_key: "92ecb16e98f0351de17886fc99b0db4d"
server_key_iv: "40415a3fd80c874f6c7d24b2370f8800"
server_key_version: "encryptionkey@0"
I20260806 09:26:57.934507 10956 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "c49c706faf5540909464fb53a1064c6f"
format_stamp: "Formatted at 2026-08-06 09:26:57 on dist-test-slave-h3ft"
server_key: "92ecb16e98f0351de17886fc99b0db4d"
server_key_iv: "40415a3fd80c874f6c7d24b2370f8800"
server_key_version: "encryptionkey@0"
I20260806 09:26:57.941097 10956 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:26:57.945940 10973 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:57.947880 10956 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:26:57.948150 10956 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "c49c706faf5540909464fb53a1064c6f"
format_stamp: "Formatted at 2026-08-06 09:26:57 on dist-test-slave-h3ft"
server_key: "92ecb16e98f0351de17886fc99b0db4d"
server_key_iv: "40415a3fd80c874f6c7d24b2370f8800"
server_key_version: "encryptionkey@0"
I20260806 09:26:57.948447 10956 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:57.965399 10956 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:57.966399 10956 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:57.966766 10956 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:57.992216 10956 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:43329
I20260806 09:26:57.992276 11045 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:43329 every 8 connection(s)
I20260806 09:26:57.993870 10956 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:26:57.999295 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 10956
I20260806 09:26:57.999553 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:26:58.000294 16730 external_mini_cluster.cc:1428] Setting key b8c69b44b2da1f37cb52acd6b39af167
I20260806 09:26:58.000859 11046 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:58.020645 11050 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:58.003200 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34242 (local address 127.16.86.190:43329)
0806 09:26:58.004845 (+ 1645us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:58.004859 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:58.005036 (+ 177us) server_negotiation.cc:407] Connection header received
0806 09:26:58.007707 (+ 2671us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:58.007734 (+ 27us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:58.008358 (+ 624us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:58.008762 (+ 404us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:58.009706 (+ 944us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:58.011658 (+ 1952us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:58.013693 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:58.015067 (+ 1374us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:58.015367 (+ 300us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:58.016752 (+ 1385us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:58.016766 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:58.016793 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:58.017805 (+ 1012us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:58.017966 (+ 161us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:58.017999 (+ 33us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:58.018159 (+ 160us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:58.018452 (+ 293us) server_negotiation.cc:299] Negotiation successful
0806 09:26:58.018894 (+ 442us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":550,"threads_started":1}
I20260806 09:26:58.021102 11046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f: Bootstrap starting.
I20260806 09:26:58.025774 11046 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:58.027392 11046 log.cc:824] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:58.031598 11046 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f: No bootstrap required, opened a new log
I20260806 09:26:58.040962 11046 raft_consensus.cc:348] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49c706faf5540909464fb53a1064c6f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43329 } }
I20260806 09:26:58.041543 11046 raft_consensus.cc:374] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:58.041702 11046 raft_consensus.cc:729] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c49c706faf5540909464fb53a1064c6f, State: Initialized, Role: FOLLOWER
I20260806 09:26:58.042531 11046 consensus_queue.cc:260] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49c706faf5540909464fb53a1064c6f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43329 } }
I20260806 09:26:58.042842 11046 raft_consensus.cc:388] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:26:58.043042 11046 raft_consensus.cc:482] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:26:58.043287 11046 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:58.046015 11046 raft_consensus.cc:504] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49c706faf5540909464fb53a1064c6f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43329 } }
I20260806 09:26:58.046685 11046 leader_election.cc:302] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c49c706faf5540909464fb53a1064c6f; no voters:
I20260806 09:26:58.047533 11046 leader_election.cc:288] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:26:58.047777 11052 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:26:58.048895 11052 raft_consensus.cc:686] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [term 1 LEADER]: Becoming Leader. State: Replica: c49c706faf5540909464fb53a1064c6f, State: Running, Role: LEADER
I20260806 09:26:58.049520 11052 consensus_queue.cc:237] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49c706faf5540909464fb53a1064c6f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43329 } }
I20260806 09:26:58.050478 11046 sys_catalog.cc:564] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:26:58.053776 11054 sys_catalog.cc:455] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c49c706faf5540909464fb53a1064c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49c706faf5540909464fb53a1064c6f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43329 } } }
I20260806 09:26:58.054463 11054 sys_catalog.cc:458] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [sys.catalog]: This master's current role is: LEADER
I20260806 09:26:58.054653 11056 sys_catalog.cc:455] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [sys.catalog]: SysCatalogTable state changed. Reason: New leader c49c706faf5540909464fb53a1064c6f. Latest consensus state: current_term: 1 leader_uuid: "c49c706faf5540909464fb53a1064c6f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c49c706faf5540909464fb53a1064c6f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 43329 } } }
I20260806 09:26:58.055270 11056 sys_catalog.cc:458] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f [sys.catalog]: This master's current role is: LEADER
I20260806 09:26:58.057478 11063 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:26:58.063808 11063 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:26:58.075740 11063 catalog_manager.cc:1347] Generated new cluster ID: 1154e3a49f25486eb7c1c371e63ecdb8
I20260806 09:26:58.075891 11063 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:26:58.110288 11063 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:26:58.111862 11063 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:26:58.126178 11063 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f: Generated new TSK 0
I20260806 09:26:58.127225 11063 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:26:58.150715 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43329
--builtin_ntp_servers=127.16.86.148:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:26:58.539151 11077 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:58.539480 11077 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:58.539646 11077 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:58.547325 11077 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:58.547454 11077 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:58.547628 11077 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:26:58.558147 11077 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:58.559644 11077 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:58.561502 11077 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:58.576350 11082 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:58.576809 11087 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:58.576831 11083 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:58.578097 11077 server_base.cc:1034] running on GCE node
I20260806 09:26:58.579142 11077 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:58.580564 11077 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:58.581888 11077 hybrid_clock.cc:648] HybridClock initialized: now 1786008418581789 us; error 44 us; skew 500 ppm
I20260806 09:26:58.582360 11077 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:26:58.585767 11077 webserver.cc:460] Webserver started at http://127.16.86.129:34953/ using document root <none> and password file <none>
I20260806 09:26:58.586711 11077 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:58.586912 11077 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:58.587354 11077 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:58.590966 11077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "56c05affbcec48b597b84aab95d5a15a"
format_stamp: "Formatted at 2026-08-06 09:26:58 on dist-test-slave-h3ft"
server_key: "d61d5ad53c2a7a08d211c27f7b0d0c91"
server_key_iv: "ed32b857e8389ce665b7952e00e0ca90"
server_key_version: "encryptionkey@0"
I20260806 09:26:58.592008 11077 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "56c05affbcec48b597b84aab95d5a15a"
format_stamp: "Formatted at 2026-08-06 09:26:58 on dist-test-slave-h3ft"
server_key: "d61d5ad53c2a7a08d211c27f7b0d0c91"
server_key_iv: "ed32b857e8389ce665b7952e00e0ca90"
server_key_version: "encryptionkey@0"
I20260806 09:26:58.598865 11077 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.001s
I20260806 09:26:58.603986 11097 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:58.605949 11077 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.002s
I20260806 09:26:58.606208 11077 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "56c05affbcec48b597b84aab95d5a15a"
format_stamp: "Formatted at 2026-08-06 09:26:58 on dist-test-slave-h3ft"
server_key: "d61d5ad53c2a7a08d211c27f7b0d0c91"
server_key_iv: "ed32b857e8389ce665b7952e00e0ca90"
server_key_version: "encryptionkey@0"
I20260806 09:26:58.606509 11077 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:58.626494 11077 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:58.627635 11077 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:58.628019 11077 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:58.629226 11077 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:58.631079 11077 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:58.631222 11077 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:58.631384 11077 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:58.631484 11077 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:58.674003 11077 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:41491
I20260806 09:26:58.674072 11263 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:41491 every 8 connection(s)
I20260806 09:26:58.675810 11077 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:26:58.679412 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 11077
I20260806 09:26:58.679646 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:26:58.680191 16730 external_mini_cluster.cc:1428] Setting key fc3770ff16005022f83be855512726bb
I20260806 09:26:58.683331 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43329
--builtin_ntp_servers=127.16.86.148:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:26:58.691766 11266 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:58.679163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:50261 (local address 127.16.86.190:43329)
0806 09:26:58.679832 (+ 669us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:58.679846 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:58.683123 (+ 3277us) server_negotiation.cc:407] Connection header received
0806 09:26:58.683855 (+ 732us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:58.683862 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:58.683982 (+ 120us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:58.684127 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:58.685560 (+ 1433us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:58.686364 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:58.688256 (+ 1892us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:58.688712 (+ 456us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:58.688866 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:58.690664 (+ 1798us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:58.690670 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:58.690674 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:58.690699 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:58.690749 (+ 50us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:58.690762 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:58.690877 (+ 115us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:58.691196 (+ 319us) server_negotiation.cc:299] Negotiation successful
0806 09:26:58.691328 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":295,"threads_started":1}
I20260806 09:26:58.693145 11268 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:58.679968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43329 (local address 127.16.86.129:50261)
0806 09:26:58.681289 (+ 1321us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:58.681343 (+ 54us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:58.683474 (+ 2131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:58.684455 (+ 981us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:58.684466 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:58.684587 (+ 121us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:58.685349 (+ 762us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:58.685369 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:58.686574 (+ 1205us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:58.686579 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:58.688082 (+ 1503us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:58.688096 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:58.689023 (+ 927us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:58.689033 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:58.689302 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:58.690243 (+ 941us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:26:58.690246 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:58.690301 (+ 55us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:26:58.690420 (+ 119us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:58.690884 (+ 464us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:58.690898 (+ 14us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:58.690908 (+ 10us) client_negotiation.cc:769] Sending connection context
0806 09:26:58.691184 (+ 276us) client_negotiation.cc:240] Negotiation successful
0806 09:26:58.691706 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":647,"thread_start_us":409,"threads_started":1}
I20260806 09:26:58.695492 11264 heartbeater.cc:347] Connected to a master server at 127.16.86.190:43329
I20260806 09:26:58.695955 11264 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:58.697086 11264 heartbeater.cc:511] Master 127.16.86.190:43329 requested a full tablet report, sending...
I20260806 09:26:58.699844 10996 ts_manager.cc:194] Registered new tserver with Master: 56c05affbcec48b597b84aab95d5a15a (127.16.86.129:41491)
I20260806 09:26:58.701442 10996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.86.129:50261
W20260806 09:26:59.091302 11269 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:59.091639 11269 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:59.091802 11269 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:59.099562 11269 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:59.099699 11269 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:59.099879 11269 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:26:59.109939 11269 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:59.111490 11269 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:59.113684 11269 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:59.130721 11275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:59.130759 11277 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:59.130728 11279 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:59.131992 11269 server_base.cc:1034] running on GCE node
I20260806 09:26:59.132731 11269 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:59.133879 11269 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:59.135212 11269 hybrid_clock.cc:648] HybridClock initialized: now 1786008419135132 us; error 49 us; skew 500 ppm
I20260806 09:26:59.135667 11269 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:26:59.138820 11269 webserver.cc:460] Webserver started at http://127.16.86.130:33359/ using document root <none> and password file <none>
I20260806 09:26:59.139700 11269 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:59.139818 11269 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:59.140116 11269 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:59.143339 11269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d"
format_stamp: "Formatted at 2026-08-06 09:26:59 on dist-test-slave-h3ft"
server_key: "d044ecf009b5b192cdc2f7e12b11a014"
server_key_iv: "81a1a66975dcc9414f9936a6e62a3077"
server_key_version: "encryptionkey@0"
I20260806 09:26:59.144157 11269 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d"
format_stamp: "Formatted at 2026-08-06 09:26:59 on dist-test-slave-h3ft"
server_key: "d044ecf009b5b192cdc2f7e12b11a014"
server_key_iv: "81a1a66975dcc9414f9936a6e62a3077"
server_key_version: "encryptionkey@0"
I20260806 09:26:59.150326 11269 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260806 09:26:59.155301 11287 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:59.157099 11269 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:26:59.157380 11269 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d"
format_stamp: "Formatted at 2026-08-06 09:26:59 on dist-test-slave-h3ft"
server_key: "d044ecf009b5b192cdc2f7e12b11a014"
server_key_iv: "81a1a66975dcc9414f9936a6e62a3077"
server_key_version: "encryptionkey@0"
I20260806 09:26:59.157717 11269 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:59.187264 11269 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:59.188244 11269 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:59.188606 11269 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:59.189831 11269 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:59.191618 11269 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:59.191715 11269 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:59.191877 11269 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:59.191978 11269 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:59.233844 11269 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:41369
I20260806 09:26:59.234026 11454 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:41369 every 8 connection(s)
I20260806 09:26:59.235427 11269 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:26:59.237157 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 11269
I20260806 09:26:59.237381 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:26:59.237867 16730 external_mini_cluster.cc:1428] Setting key fa6ec6da239f9bb8e7e8ddcb013b8a3e
I20260806 09:26:59.243183 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43329
--builtin_ntp_servers=127.16.86.148:45137
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:26:59.251279 11457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.239309 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:53727 (local address 127.16.86.190:43329)
0806 09:26:59.241551 (+ 2242us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:59.241557 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:59.241616 (+ 59us) server_negotiation.cc:407] Connection header received
0806 09:26:59.243902 (+ 2286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:59.243911 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:59.244009 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:59.244152 (+ 143us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:59.245331 (+ 1179us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.246107 (+ 776us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.247802 (+ 1695us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.248243 (+ 441us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.248391 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.250172 (+ 1781us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:59.250180 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:59.250186 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:59.250223 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:59.250272 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:59.250286 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:59.250406 (+ 120us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:59.250679 (+ 273us) server_negotiation.cc:299] Negotiation successful
0806 09:26:59.250802 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2114,"thread_start_us":1986,"threads_started":1}
I20260806 09:26:59.252609 11459 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.239815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43329 (local address 127.16.86.130:53727)
0806 09:26:59.241440 (+ 1625us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:59.241512 (+ 72us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:59.243528 (+ 2016us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:59.244415 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:59.244424 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:59.244521 (+ 97us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:59.245149 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.245167 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.246275 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.246285 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.247647 (+ 1362us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.247660 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.248503 (+ 843us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.248511 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.248763 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.249753 (+ 990us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:26:59.249757 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:59.249802 (+ 45us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:26:59.249900 (+ 98us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:59.250407 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:59.250422 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:59.250434 (+ 12us) client_negotiation.cc:769] Sending connection context
0806 09:26:59.250668 (+ 234us) client_negotiation.cc:240] Negotiation successful
0806 09:26:59.251173 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"thread_start_us":410,"threads_started":1}
I20260806 09:26:59.254516 11455 heartbeater.cc:347] Connected to a master server at 127.16.86.190:43329
I20260806 09:26:59.255069 11455 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:59.256361 11455 heartbeater.cc:511] Master 127.16.86.190:43329 requested a full tablet report, sending...
I20260806 09:26:59.258368 10996 ts_manager.cc:194] Registered new tserver with Master: 0f7a9d9f3d3f412a87b09c9a2d25117d (127.16.86.130:41369)
I20260806 09:26:59.259358 10996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.86.130:53727
W20260806 09:26:59.610131 11460 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:26:59.610430 11460 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:26:59.610518 11460 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:26:59.618217 11460 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:26:59.618343 11460 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:26:59.618453 11460 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:26:59.627694 11460 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:45137
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:43329
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:26:59.628966 11460 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:26:59.630805 11460 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:26:59.645368 11466 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:59.645332 11467 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:26:59.645552 11471 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:26:59.646863 11460 server_base.cc:1034] running on GCE node
I20260806 09:26:59.647899 11460 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:26:59.649087 11460 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:26:59.650414 11460 hybrid_clock.cc:648] HybridClock initialized: now 1786008419650319 us; error 72 us; skew 500 ppm
I20260806 09:26:59.651068 11460 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:26:59.654599 11460 webserver.cc:460] Webserver started at http://127.16.86.131:38223/ using document root <none> and password file <none>
I20260806 09:26:59.655632 11460 fs_manager.cc:359] Metadata directory not provided
I20260806 09:26:59.655792 11460 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:26:59.656209 11460 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:26:59.660334 11460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "a65cb9ca52084e0e976050cf87c52592"
format_stamp: "Formatted at 2026-08-06 09:26:59 on dist-test-slave-h3ft"
server_key: "0d9ab5334919f323ed386daa1f257a12"
server_key_iv: "1d14bb6585d74e4d9937e07822bca0a3"
server_key_version: "encryptionkey@0"
I20260806 09:26:59.661401 11460 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "a65cb9ca52084e0e976050cf87c52592"
format_stamp: "Formatted at 2026-08-06 09:26:59 on dist-test-slave-h3ft"
server_key: "0d9ab5334919f323ed386daa1f257a12"
server_key_iv: "1d14bb6585d74e4d9937e07822bca0a3"
server_key_version: "encryptionkey@0"
I20260806 09:26:59.668972 11460 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260806 09:26:59.674211 11479 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:59.676494 11460 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260806 09:26:59.676726 11460 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "a65cb9ca52084e0e976050cf87c52592"
format_stamp: "Formatted at 2026-08-06 09:26:59 on dist-test-slave-h3ft"
server_key: "0d9ab5334919f323ed386daa1f257a12"
server_key_iv: "1d14bb6585d74e4d9937e07822bca0a3"
server_key_version: "encryptionkey@0"
I20260806 09:26:59.677045 11460 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:26:59.705027 11264 heartbeater.cc:503] Master 127.16.86.190:43329 was elected leader, sending a full tablet report...
I20260806 09:26:59.709334 11460 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:26:59.710579 11460 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:26:59.711084 11460 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:26:59.712666 11460 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:26:59.714759 11460 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:26:59.714895 11460 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:59.715150 11460 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:26:59.715266 11460 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:26:59.760140 11460 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:41709
I20260806 09:26:59.760275 11635 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:41709 every 8 connection(s)
I20260806 09:26:59.761909 11460 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:26:59.767414 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 11460
I20260806 09:26:59.767656 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:26:59.768378 16730 external_mini_cluster.cc:1428] Setting key 27b09f196333d909c7124780350f5038
I20260806 09:26:59.778940 11639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.765581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:36685 (local address 127.16.86.190:43329)
0806 09:26:59.766149 (+ 568us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:59.766155 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:59.768075 (+ 1920us) server_negotiation.cc:407] Connection header received
0806 09:26:59.770951 (+ 2876us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:59.770955 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:59.771119 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:59.771237 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:59.773003 (+ 1766us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.773869 (+ 866us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.775815 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.776233 (+ 418us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.776344 (+ 111us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.777988 (+ 1644us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:59.777994 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:59.777997 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:59.778018 (+ 21us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:59.778064 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:59.778076 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:59.778168 (+ 92us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:59.778548 (+ 380us) server_negotiation.cc:299] Negotiation successful
0806 09:26:59.778639 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":425,"thread_start_us":273,"threads_started":1}
I20260806 09:26:59.780580 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.766254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:43329 (local address 127.16.86.131:36685)
0806 09:26:59.767804 (+ 1550us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:59.767893 (+ 89us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:59.770534 (+ 2641us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:59.771666 (+ 1132us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:59.771681 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:59.771847 (+ 166us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:26:59.772753 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.772804 (+ 51us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.774046 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.774051 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.775652 (+ 1601us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.775668 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.776491 (+ 823us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.776496 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.776742 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.777600 (+ 858us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:26:59.777603 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:26:59.777661 (+ 58us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:26:59.777769 (+ 108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:26:59.778239 (+ 470us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:26:59.778251 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:26:59.778260 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 09:26:59.778518 (+ 258us) client_negotiation.cc:240] Negotiation successful
0806 09:26:59.778955 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":681,"thread_start_us":376,"threads_started":1}
I20260806 09:26:59.783070 11637 heartbeater.cc:347] Connected to a master server at 127.16.86.190:43329
I20260806 09:26:59.783455 11637 heartbeater.cc:464] Registering TS with master...
I20260806 09:26:59.784567 11637 heartbeater.cc:511] Master 127.16.86.190:43329 requested a full tablet report, sending...
I20260806 09:26:59.786237 10996 ts_manager.cc:194] Registered new tserver with Master: a65cb9ca52084e0e976050cf87c52592 (127.16.86.131:41709)
I20260806 09:26:59.787134 10996 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.86.131:36685
I20260806 09:26:59.799387 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:26:59.812608 11639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.805243 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46518 (local address 127.16.86.190:43329)
0806 09:26:59.805625 (+ 382us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:59.805632 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:59.805995 (+ 363us) server_negotiation.cc:407] Connection header received
0806 09:26:59.806182 (+ 187us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:59.806186 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:59.806259 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:59.806363 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:26:59.807126 (+ 763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.807864 (+ 738us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.809563 (+ 1699us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.809974 (+ 411us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.810110 (+ 136us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.811597 (+ 1487us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:26:59.811603 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:26:59.811606 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:26:59.811633 (+ 27us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:26:59.811675 (+ 42us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:26:59.811688 (+ 13us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:26:59.811872 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:59.812172 (+ 300us) server_negotiation.cc:299] Negotiation successful
0806 09:26:59.812243 (+ 71us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260806 09:26:59.817875 10996 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:46518:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:26:59.825932 10996 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:26:59.867260 11654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.842323 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:41491 (local address 127.0.0.1:57084)
0806 09:26:59.856192 (+ 13869us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:59.856269 (+ 77us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:59.856580 (+ 311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:59.858387 (+ 1807us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:59.858395 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:59.858427 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:59.858955 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.858975 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.861016 (+ 2041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.861020 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.863647 (+ 2627us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.863661 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.866181 (+ 2520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.866188 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.866387 (+ 199us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.866450 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 09:26:59.866594 (+ 144us) client_negotiation.cc:240] Negotiation successful
0806 09:26:59.866848 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13345,"thread_start_us":15055,"threads_started":1}
I20260806 09:26:59.867980 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.842432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57084 (local address 127.16.86.129:41491)
0806 09:26:59.857377 (+ 14945us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:59.857400 (+ 23us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:59.857441 (+ 41us) server_negotiation.cc:407] Connection header received
0806 09:26:59.857562 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:59.857570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:59.857850 (+ 280us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:59.858055 (+ 205us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:59.859228 (+ 1173us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.860831 (+ 1603us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.863909 (+ 3078us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.865479 (+ 1570us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.865626 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.866032 (+ 406us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:59.867400 (+ 1368us) server_negotiation.cc:299] Negotiation successful
0806 09:26:59.867607 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14754,"thread_start_us":14601,"threads_started":1}
I20260806 09:26:59.873486 11660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.842421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:41709 (local address 127.0.0.1:59386)
0806 09:26:59.859483 (+ 17062us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:59.859503 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:59.859623 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:59.862717 (+ 3094us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:59.862722 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:59.862748 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:59.863241 (+ 493us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.863254 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.867508 (+ 4254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.867515 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.870402 (+ 2887us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.870417 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.872387 (+ 1970us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.872395 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.872673 (+ 278us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.872716 (+ 43us) client_negotiation.cc:769] Sending connection context
0806 09:26:59.872849 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 09:26:59.873054 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16909,"thread_start_us":24166,"threads_started":1}
I20260806 09:26:59.873495 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.843211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59386 (local address 127.16.86.131:41709)
0806 09:26:59.860288 (+ 17077us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:59.860297 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:59.860331 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:26:59.860434 (+ 103us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:59.860442 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:59.860648 (+ 206us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:59.862406 (+ 1758us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:59.863477 (+ 1071us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.867176 (+ 3699us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.870621 (+ 3445us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.872082 (+ 1461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.872243 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.872532 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:59.872923 (+ 391us) server_negotiation.cc:299] Negotiation successful
0806 09:26:59.873105 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16857,"thread_start_us":16695,"threads_started":1}
I20260806 09:26:59.873286 11169 tablet_service.cc:1467] Processing CreateTablet for tablet f65b621e33234b9eb925ffeb9badd7ec (DEFAULT_TABLE table=test-table [id=e9b4ba5cabbe4c7e9d1fa146005db8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:59.875660 11169 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65b621e33234b9eb925ffeb9badd7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:59.876499 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.843823 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:41369 (local address 127.0.0.1:34478)
0806 09:26:59.866910 (+ 23087us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:59.866931 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:59.867096 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:59.868753 (+ 1657us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:59.868760 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:59.868802 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:59.869264 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.869278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.871458 (+ 2180us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.871464 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.873570 (+ 2106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.873588 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.875559 (+ 1971us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.875566 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.875864 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.875905 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:26:59.876029 (+ 124us) client_negotiation.cc:240] Negotiation successful
0806 09:26:59.876172 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":22942,"thread_start_us":15311,"threads_started":1}
I20260806 09:26:59.876600 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.842374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34478 (local address 127.16.86.130:41369)
0806 09:26:59.866381 (+ 24007us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:59.866390 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:59.867127 (+ 737us) server_negotiation.cc:407] Connection header received
0806 09:26:59.867300 (+ 173us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:59.867314 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:59.868171 (+ 857us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:59.868440 (+ 269us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:59.869520 (+ 1080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.871271 (+ 1751us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.873865 (+ 2594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.875338 (+ 1473us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.875504 (+ 166us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.875759 (+ 255us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:59.876077 (+ 318us) server_negotiation.cc:299] Negotiation successful
0806 09:26:59.876256 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":23816,"thread_start_us":23668,"threads_started":1}
I20260806 09:26:59.880534 11546 tablet_service.cc:1467] Processing CreateTablet for tablet f65b621e33234b9eb925ffeb9badd7ec (DEFAULT_TABLE table=test-table [id=e9b4ba5cabbe4c7e9d1fa146005db8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:59.881373 11356 tablet_service.cc:1467] Processing CreateTablet for tablet f65b621e33234b9eb925ffeb9badd7ec (DEFAULT_TABLE table=test-table [id=e9b4ba5cabbe4c7e9d1fa146005db8c4]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:26:59.882516 11546 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65b621e33234b9eb925ffeb9badd7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:59.883248 11356 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet f65b621e33234b9eb925ffeb9badd7ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:26:59.894815 11661 tablet_bootstrap.cc:492] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592: Bootstrap starting.
I20260806 09:26:59.898859 11661 tablet_bootstrap.cc:654] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:59.899926 11661 log.cc:824] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:59.902491 11669 tablet_bootstrap.cc:492] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d: Bootstrap starting.
I20260806 09:26:59.903330 11666 tablet_bootstrap.cc:492] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a: Bootstrap starting.
I20260806 09:26:59.903591 11661 tablet_bootstrap.cc:492] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592: No bootstrap required, opened a new log
I20260806 09:26:59.903892 11661 ts_tablet_manager.cc:1397] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20260806 09:26:59.906641 11669 tablet_bootstrap.cc:654] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:59.907210 11666 tablet_bootstrap.cc:654] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a: Neither blocks nor log segments found. Creating new log.
I20260806 09:26:59.908027 11669 log.cc:824] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:59.908739 11666 log.cc:824] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:26:59.914698 11669 tablet_bootstrap.cc:492] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d: No bootstrap required, opened a new log
I20260806 09:26:59.915094 11669 ts_tablet_manager.cc:1397] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.009s
I20260806 09:26:59.915810 11661 raft_consensus.cc:348] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c05affbcec48b597b84aab95d5a15a" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41491 } } peers { permanent_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41369 } } peers { permanent_uuid: "a65cb9ca52084e0e976050cf87c52592" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 41709 } }
I20260806 09:26:59.916322 11661 raft_consensus.cc:374] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:59.916437 11661 raft_consensus.cc:729] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a65cb9ca52084e0e976050cf87c52592, State: Initialized, Role: FOLLOWER
I20260806 09:26:59.917161 11661 consensus_queue.cc:260] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c05affbcec48b597b84aab95d5a15a" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41491 } } peers { permanent_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41369 } } peers { permanent_uuid: "a65cb9ca52084e0e976050cf87c52592" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 41709 } }
I20260806 09:26:59.918602 11666 tablet_bootstrap.cc:492] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a: No bootstrap required, opened a new log
I20260806 09:26:59.918628 11637 heartbeater.cc:503] Master 127.16.86.190:43329 was elected leader, sending a full tablet report...
I20260806 09:26:59.919030 11666 ts_tablet_manager.cc:1397] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a: Time spent bootstrapping tablet: real 0.018s user 0.003s sys 0.010s
I20260806 09:26:59.919742 11661 ts_tablet_manager.cc:1428] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592: Time spent starting tablet: real 0.016s user 0.016s sys 0.000s
I20260806 09:26:59.925565 11669 raft_consensus.cc:348] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c05affbcec48b597b84aab95d5a15a" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41491 } } peers { permanent_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41369 } } peers { permanent_uuid: "a65cb9ca52084e0e976050cf87c52592" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 41709 } }
I20260806 09:26:59.926064 11669 raft_consensus.cc:374] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:59.926211 11669 raft_consensus.cc:729] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0f7a9d9f3d3f412a87b09c9a2d25117d, State: Initialized, Role: FOLLOWER
I20260806 09:26:59.926821 11669 consensus_queue.cc:260] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c05affbcec48b597b84aab95d5a15a" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41491 } } peers { permanent_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41369 } } peers { permanent_uuid: "a65cb9ca52084e0e976050cf87c52592" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 41709 } }
I20260806 09:26:59.928489 11455 heartbeater.cc:503] Master 127.16.86.190:43329 was elected leader, sending a full tablet report...
I20260806 09:26:59.928958 11669 ts_tablet_manager.cc:1428] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d: Time spent starting tablet: real 0.014s user 0.008s sys 0.005s
I20260806 09:26:59.929840 11666 raft_consensus.cc:348] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c05affbcec48b597b84aab95d5a15a" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41491 } } peers { permanent_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41369 } } peers { permanent_uuid: "a65cb9ca52084e0e976050cf87c52592" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 41709 } }
I20260806 09:26:59.930455 11666 raft_consensus.cc:374] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:26:59.930588 11666 raft_consensus.cc:729] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 56c05affbcec48b597b84aab95d5a15a, State: Initialized, Role: FOLLOWER
I20260806 09:26:59.931432 11666 consensus_queue.cc:260] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c05affbcec48b597b84aab95d5a15a" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41491 } } peers { permanent_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41369 } } peers { permanent_uuid: "a65cb9ca52084e0e976050cf87c52592" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 41709 } }
I20260806 09:26:59.933666 11666 ts_tablet_manager.cc:1428] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a: Time spent starting tablet: real 0.014s user 0.011s sys 0.002s
I20260806 09:26:59.941740 11672 raft_consensus.cc:482] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:26:59.942104 11672 raft_consensus.cc:504] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c05affbcec48b597b84aab95d5a15a" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41491 } } peers { permanent_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41369 } } peers { permanent_uuid: "a65cb9ca52084e0e976050cf87c52592" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 41709 } }
I20260806 09:26:59.944005 11672 leader_election.cc:288] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 56c05affbcec48b597b84aab95d5a15a (127.16.86.129:41491), 0f7a9d9f3d3f412a87b09c9a2d25117d (127.16.86.130:41369)
I20260806 09:26:59.952806 11641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.944530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:41369 (local address 127.16.86.131:44983)
0806 09:26:59.944926 (+ 396us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:59.944949 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:59.945117 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:59.945782 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:59.945787 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:59.945817 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:59.946048 (+ 231us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.946063 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.947675 (+ 1612us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.947678 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.949891 (+ 2213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.949911 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.951877 (+ 1966us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.951883 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.952015 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.952044 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 09:26:59.952138 (+ 94us) client_negotiation.cc:240] Negotiation successful
0806 09:26:59.952392 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":241,"mutex_wait_us":4}
I20260806 09:26:59.952929 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.944520 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:44983 (local address 127.16.86.130:41369)
0806 09:26:59.944735 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:59.944743 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:59.945094 (+ 351us) server_negotiation.cc:407] Connection header received
0806 09:26:59.945344 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:59.945351 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:59.945516 (+ 165us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:59.945663 (+ 147us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:59.946273 (+ 610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.947472 (+ 1199us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.950145 (+ 2673us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.951642 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.951854 (+ 212us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.951958 (+ 104us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:59.952267 (+ 309us) server_negotiation.cc:299] Negotiation successful
0806 09:26:59.952562 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20260806 09:26:59.953229 11676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.944837 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:41491 (local address 127.16.86.131:41571)
0806 09:26:59.945840 (+ 1003us) negotiation.cc:107] Waiting for socket to connect
0806 09:26:59.945870 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:26:59.946006 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:26:59.946479 (+ 473us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:26:59.946482 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:26:59.946502 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:26:59.946809 (+ 307us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.946818 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.948191 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.948197 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.950253 (+ 2056us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:26:59.950263 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.952354 (+ 2091us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.952361 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:26:59.952514 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.952562 (+ 48us) client_negotiation.cc:769] Sending connection context
0806 09:26:59.952662 (+ 100us) client_negotiation.cc:240] Negotiation successful
0806 09:26:59.952786 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":868,"mutex_wait_us":66,"thread_start_us":596,"threads_started":1}
I20260806 09:26:59.953790 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.944807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:41571 (local address 127.16.86.129:41491)
0806 09:26:59.945103 (+ 296us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:59.945110 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:59.946029 (+ 919us) server_negotiation.cc:407] Connection header received
0806 09:26:59.946133 (+ 104us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:59.946137 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:59.946315 (+ 178us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:59.946458 (+ 143us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:26:59.946991 (+ 533us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.947988 (+ 997us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.950470 (+ 2482us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.952095 (+ 1625us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:26:59.952339 (+ 244us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:26:59.952417 (+ 78us) server_negotiation.cc:1035] Waiting for connection context
0806 09:26:59.952742 (+ 325us) server_negotiation.cc:299] Negotiation successful
0806 09:26:59.952912 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260806 09:26:59.954380 11388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65b621e33234b9eb925ffeb9badd7ec" candidate_uuid: "a65cb9ca52084e0e976050cf87c52592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d" is_pre_election: true
I20260806 09:26:59.955242 11388 raft_consensus.cc:2454] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a65cb9ca52084e0e976050cf87c52592 in term 0.
I20260806 09:26:59.955494 11198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65b621e33234b9eb925ffeb9badd7ec" candidate_uuid: "a65cb9ca52084e0e976050cf87c52592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c05affbcec48b597b84aab95d5a15a" is_pre_election: true
I20260806 09:26:59.956269 11198 raft_consensus.cc:2454] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a65cb9ca52084e0e976050cf87c52592 in term 0.
I20260806 09:26:59.956584 11481 leader_election.cc:302] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0f7a9d9f3d3f412a87b09c9a2d25117d, a65cb9ca52084e0e976050cf87c52592; no voters:
I20260806 09:26:59.957152 11672 raft_consensus.cc:2781] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:26:59.957347 11672 raft_consensus.cc:482] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:26:59.957494 11672 raft_consensus.cc:3037] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:59.960026 11672 raft_consensus.cc:504] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c05affbcec48b597b84aab95d5a15a" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41491 } } peers { permanent_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41369 } } peers { permanent_uuid: "a65cb9ca52084e0e976050cf87c52592" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 41709 } }
I20260806 09:26:59.960937 11672 leader_election.cc:288] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [CANDIDATE]: Term 1 election: Requested vote from peers 56c05affbcec48b597b84aab95d5a15a (127.16.86.129:41491), 0f7a9d9f3d3f412a87b09c9a2d25117d (127.16.86.130:41369)
I20260806 09:26:59.961845 11198 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65b621e33234b9eb925ffeb9badd7ec" candidate_uuid: "a65cb9ca52084e0e976050cf87c52592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "56c05affbcec48b597b84aab95d5a15a"
I20260806 09:26:59.962157 11198 raft_consensus.cc:3037] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:59.962370 11388 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "f65b621e33234b9eb925ffeb9badd7ec" candidate_uuid: "a65cb9ca52084e0e976050cf87c52592" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d"
I20260806 09:26:59.962693 11388 raft_consensus.cc:3037] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:26:59.964978 11198 raft_consensus.cc:2454] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a65cb9ca52084e0e976050cf87c52592 in term 1.
I20260806 09:26:59.965742 11482 leader_election.cc:302] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 56c05affbcec48b597b84aab95d5a15a, a65cb9ca52084e0e976050cf87c52592; no voters:
I20260806 09:26:59.966190 11672 raft_consensus.cc:2781] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:26:59.966215 11388 raft_consensus.cc:2454] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a65cb9ca52084e0e976050cf87c52592 in term 1.
I20260806 09:26:59.966846 11672 raft_consensus.cc:686] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 1 LEADER]: Becoming Leader. State: Replica: a65cb9ca52084e0e976050cf87c52592, State: Running, Role: LEADER
I20260806 09:26:59.967763 11672 consensus_queue.cc:237] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c05affbcec48b597b84aab95d5a15a" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41491 } } peers { permanent_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41369 } } peers { permanent_uuid: "a65cb9ca52084e0e976050cf87c52592" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 41709 } }
I20260806 09:26:59.974390 10996 catalog_manager.cc:5495] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 reported cstate change: term changed from 0 to 1, leader changed from <none> to a65cb9ca52084e0e976050cf87c52592 (127.16.86.131). New cstate: current_term: 1 leader_uuid: "a65cb9ca52084e0e976050cf87c52592" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "56c05affbcec48b597b84aab95d5a15a" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41491 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41369 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a65cb9ca52084e0e976050cf87c52592" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 41709 } health_report { overall_health: HEALTHY } } }
W20260806 09:26:59.988044 11456 tablet.cc:2369] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:27:00.004716 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:26:59.996128 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59394 (local address 127.16.86.131:41709)
0806 09:26:59.996405 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 09:26:59.996413 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:26:59.996645 (+ 232us) server_negotiation.cc:407] Connection header received
0806 09:26:59.996964 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:26:59.996969 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:26:59.997073 (+ 104us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:26:59.997275 (+ 202us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:26:59.997772 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:26:59.998633 (+ 861us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:00.001158 (+ 2525us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:00.001917 (+ 759us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:00.002093 (+ 176us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:00.002959 (+ 866us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:27:00.003800 (+ 841us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:27:00.004058 (+ 258us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:00.004237 (+ 179us) server_negotiation.cc:299] Negotiation successful
0806 09:27:00.004301 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
W20260806 09:27:00.014654 11638 tablet.cc:2369] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:27:00.016695 11198 raft_consensus.cc:1261] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a [term 1 FOLLOWER]: Refusing update from remote peer a65cb9ca52084e0e976050cf87c52592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:27:00.017005 11388 raft_consensus.cc:1261] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d [term 1 FOLLOWER]: Refusing update from remote peer a65cb9ca52084e0e976050cf87c52592: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:27:00.017916 11677 consensus_queue.cc:1035] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "56c05affbcec48b597b84aab95d5a15a" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 41491 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:27:00.018447 11672 consensus_queue.cc:1035] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0f7a9d9f3d3f412a87b09c9a2d25117d" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 41369 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:27:00.052637 10993 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:46518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:27:00.052999 10993 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:46518:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:27:00.058265 10993 catalog_manager.cc:5782] T 00000000000000000000000000000000 P c49c706faf5540909464fb53a1064c6f: Sending DeleteTablet for 3 replicas of tablet f65b621e33234b9eb925ffeb9badd7ec
I20260806 09:27:00.060065 11546 tablet_service.cc:1514] Processing DeleteTablet for tablet f65b621e33234b9eb925ffeb9badd7ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:27:00 UTC) from {username='slave'} at 127.0.0.1:59386
I20260806 09:27:00.060077 11169 tablet_service.cc:1514] Processing DeleteTablet for tablet f65b621e33234b9eb925ffeb9badd7ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:27:00 UTC) from {username='slave'} at 127.0.0.1:57084
I20260806 09:27:00.060938 11356 tablet_service.cc:1514] Processing DeleteTablet for tablet f65b621e33234b9eb925ffeb9badd7ec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:27:00 UTC) from {username='slave'} at 127.0.0.1:34478
I20260806 09:27:00.061558 11691 tablet_replica.cc:330] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592: stopping tablet replica
I20260806 09:27:00.062449 11691 raft_consensus.cc:2229] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:27:00.063553 11691 raft_consensus.cc:2258] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:27:00.066900 11691 ts_tablet_manager.cc:1905] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:27:00.067171 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 11077
I20260806 09:27:00.077693 11691 ts_tablet_manager.cc:1918] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:27:00.078220 11691 log.cc:1196] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/f65b621e33234b9eb925ffeb9badd7ec
I20260806 09:27:00.079160 11691 ts_tablet_manager.cc:1939] T f65b621e33234b9eb925ffeb9badd7ec P a65cb9ca52084e0e976050cf87c52592: Deleting consensus metadata
I20260806 09:27:00.082224 10980 catalog_manager.cc:4841] TS a65cb9ca52084e0e976050cf87c52592 (127.16.86.131:41709): tablet f65b621e33234b9eb925ffeb9badd7ec (table test-table [id=e9b4ba5cabbe4c7e9d1fa146005db8c4]) successfully deleted
I20260806 09:27:00.091817 11693 tablet_replica.cc:330] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d: stopping tablet replica
I20260806 09:27:00.092762 11692 tablet_replica.cc:330] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a: stopping tablet replica
I20260806 09:27:00.093101 11693 raft_consensus.cc:2229] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:27:00.093429 11692 raft_consensus.cc:2229] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:27:00.093870 11693 raft_consensus.cc:2258] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:27:00.094125 11692 raft_consensus.cc:2258] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:27:00.096926 11692 ts_tablet_manager.cc:1905] T f65b621e33234b9eb925ffeb9badd7ec P 56c05affbcec48b597b84aab95d5a15a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:27:00.096913 11693 ts_tablet_manager.cc:1905] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:27:00.101310 11657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:00.069611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57088 (local address 127.16.86.129:41491)
0806 09:27:00.069846 (+ 235us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:00.069854 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:00.091754 (+ 21900us) server_negotiation.cc:407] Connection header received
0806 09:27:00.092015 (+ 261us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:00.092022 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:00.092174 (+ 152us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:00.092362 (+ 188us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:00.093231 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:00.094546 (+ 1315us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:00.096668 (+ 2122us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:00.097370 (+ 702us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:00.097480 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:00.099283 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:00.099297 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:00.099320 (+ 23us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:00.100228 (+ 908us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:00.100306 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:00.100335 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:00.100442 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:00.100727 (+ 285us) server_negotiation.cc:299] Negotiation successful
0806 09:27:00.100865 (+ 138us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:27:00.103084 11255 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:00.107918 11693 ts_tablet_manager.cc:1918] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:27:00.108204 11693 log.cc:1196] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/f65b621e33234b9eb925ffeb9badd7ec
I20260806 09:27:00.108932 11693 ts_tablet_manager.cc:1939] T f65b621e33234b9eb925ffeb9badd7ec P 0f7a9d9f3d3f412a87b09c9a2d25117d: Deleting consensus metadata
I20260806 09:27:00.111708 10974 catalog_manager.cc:4841] TS 0f7a9d9f3d3f412a87b09c9a2d25117d (127.16.86.130:41369): tablet f65b621e33234b9eb925ffeb9badd7ec (table test-table [id=e9b4ba5cabbe4c7e9d1fa146005db8c4]) successfully deleted
I20260806 09:27:00.221997 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 11077
W20260806 09:27:00.259316 10975 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.16.86.129:41491 (error 108)
I20260806 09:27:00.259685 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 11269
I20260806 09:27:00.263707 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:00.262933 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:41491 (local address 127.0.0.1:57092)
0806 09:27:00.263292 (+ 359us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:00.263469 (+ 177us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:41491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":161}
W20260806 09:27:00.264072 10975 catalog_manager.cc:4584] TS 56c05affbcec48b597b84aab95d5a15a (127.16.86.129:41491): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f65b621e33234b9eb925ffeb9badd7ec: Network error: Client connection negotiation failed: client connection to 127.16.86.129:41491: connect: Connection refused (error 111)
I20260806 09:27:00.270391 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:00.261163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34482 (local address 127.16.86.130:41369)
0806 09:27:00.261422 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:00.261430 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:00.262947 (+ 1517us) server_negotiation.cc:407] Connection header received
0806 09:27:00.263295 (+ 348us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:00.263301 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:00.263427 (+ 126us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:00.263563 (+ 136us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:00.264281 (+ 718us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:00.265162 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:00.266765 (+ 1603us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:00.267321 (+ 556us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:00.267479 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:00.268809 (+ 1330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:00.268819 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:00.268835 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:00.269500 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:00.269559 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:00.269581 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:00.269726 (+ 145us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:00.269876 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 09:27:00.270005 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20260806 09:27:00.271266 11449 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:00.318233 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:00.317494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:41491 (local address 127.0.0.1:57100)
0806 09:27:00.317839 (+ 345us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:00.317991 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:41491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":130}
I20260806 09:27:00.320133 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:00.319578 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:41491 (local address 127.0.0.1:57112)
0806 09:27:00.319803 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:00.319916 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:41491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20260806 09:27:00.369699 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 11269
I20260806 09:27:00.381101 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:00.380555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:41491 (local address 127.0.0.1:57124)
0806 09:27:00.380819 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:00.380923 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:41491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260806 09:27:00.382231 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:00.381854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:41491 (local address 127.0.0.1:57126)
0806 09:27:00.381998 (+ 144us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:00.382073 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:41491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20260806 09:27:00.408605 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 11460
I20260806 09:27:00.418710 11659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:00.409688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59408 (local address 127.16.86.131:41709)
0806 09:27:00.409903 (+ 215us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:00.409908 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:00.410007 (+ 99us) server_negotiation.cc:407] Connection header received
0806 09:27:00.410296 (+ 289us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:00.410301 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:00.410431 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:00.410542 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:00.411188 (+ 646us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:00.412189 (+ 1001us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:00.414388 (+ 2199us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:00.414970 (+ 582us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:00.415325 (+ 355us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:00.416834 (+ 1509us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:00.416843 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:00.416859 (+ 16us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:00.417653 (+ 794us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:00.417712 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:00.417735 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:00.417853 (+ 118us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:00.418059 (+ 206us) server_negotiation.cc:299] Negotiation successful
0806 09:27:00.418165 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 09:27:00.419811 11630 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:00.462992 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:00.462179 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:41491 (local address 127.0.0.1:57142)
0806 09:27:00.462555 (+ 376us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:00.462707 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:41491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":147}
I20260806 09:27:00.464686 11658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:00.464168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:41491 (local address 127.0.0.1:57156)
0806 09:27:00.464454 (+ 286us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:00.464541 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.129:41491: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":140}
I20260806 09:27:00.555390 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 11460
I20260806 09:27:00.586081 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 10956
I20260806 09:27:00.587530 11039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:00.718466 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 10956
2026-08-06T09:27:00Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (3404 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:00 dist-test-slave-h3ft krb5kdc[11703](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:00 dist-test-slave-h3ft krb5kdc[11703](info): set up 2 sockets
Aug 06 09:27:00 dist-test-slave-h3ft krb5kdc[11703](info): commencing operation
krb5kdc: starting...
W20260806 09:27:02.134802 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.361s user 0.001s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:02 dist-test-slave-h3ft krb5kdc[11703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:27:02.564719 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:40301
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:40301
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:02.988435 11719 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:02.988940 11719 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:02.989138 11719 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:02.998598 11719 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:02.998816 11719 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:02.998948 11719 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:02.999096 11719 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:02.999202 11719 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:03.011871 11719 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:40301
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:40301
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:03.013921 11719 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:03.015664 11719 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:03.031101 11725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:03.031121 11726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:03.031471 11730 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:03.033524 11719 server_base.cc:1034] running on GCE node
I20260806 09:27:03.034662 11719 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:03.036680 11719 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:03.038012 11719 hybrid_clock.cc:648] HybridClock initialized: now 1786008423037925 us; error 70 us; skew 500 ppm
I20260806 09:27:03.038549 11719 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:03 dist-test-slave-h3ft krb5kdc[11703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008423, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:03.043295 11719 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:27:03.045694 11719 webserver.cc:460] Webserver started at http://127.16.86.190:46083/ using document root <none> and password file <none>
I20260806 09:27:03.046697 11719 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:03.046901 11719 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:03.047464 11719 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:03.051651 11719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "7ec448410f3946b6a7a915adb296ad1e"
format_stamp: "Formatted at 2026-08-06 09:27:03 on dist-test-slave-h3ft"
server_key: "26a987329a76ffd85b2b9cc2642449a3"
server_key_iv: "c0a6859540584f8f04205904a6c8b0aa"
server_key_version: "encryptionkey@0"
I20260806 09:27:03.052750 11719 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "7ec448410f3946b6a7a915adb296ad1e"
format_stamp: "Formatted at 2026-08-06 09:27:03 on dist-test-slave-h3ft"
server_key: "26a987329a76ffd85b2b9cc2642449a3"
server_key_iv: "c0a6859540584f8f04205904a6c8b0aa"
server_key_version: "encryptionkey@0"
I20260806 09:27:03.060501 11719 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260806 09:27:03.065881 11738 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:03.068473 11719 fs_manager.cc:725] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260806 09:27:03.068802 11719 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "7ec448410f3946b6a7a915adb296ad1e"
format_stamp: "Formatted at 2026-08-06 09:27:03 on dist-test-slave-h3ft"
server_key: "26a987329a76ffd85b2b9cc2642449a3"
server_key_iv: "c0a6859540584f8f04205904a6c8b0aa"
server_key_version: "encryptionkey@0"
I20260806 09:27:03.069139 11719 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:03.089838 11719 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:03.096424 11719 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:03.096771 11719 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:03.123679 11719 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:40301
I20260806 09:27:03.123792 11812 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:40301 every 8 connection(s)
I20260806 09:27:03.125656 11719 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:03.129781 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 11719
I20260806 09:27:03.130007 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:03.130580 16730 external_mini_cluster.cc:1428] Setting key 0c83ad18b05cd5f27101b6e84e0e6389
I20260806 09:27:03.130976 11813 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:03.150645 11813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e: Bootstrap starting.
Aug 06 09:27:03 dist-test-slave-h3ft krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:27:03.154896 11813 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:03.156736 11813 log.cc:824] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:03.160526 11813 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e: No bootstrap required, opened a new log
I20260806 09:27:03.162412 11818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:03.133126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58984 (local address 127.16.86.190:40301)
0806 09:27:03.134202 (+ 1076us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:03.134217 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:03.134281 (+ 64us) server_negotiation.cc:407] Connection header received
0806 09:27:03.136692 (+ 2411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:03.136728 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:03.137323 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:03.137825 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:03.140402 (+ 2577us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:03.142117 (+ 1715us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:03.144600 (+ 2483us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:03.145605 (+ 1005us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:03.145822 (+ 217us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:03.152708 (+ 6886us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:03.152721 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:03.152748 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:03.152814 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:03.156687 (+ 3873us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:03.157844 (+ 1157us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:03.157853 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:03.157862 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:03.158102 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:03.158607 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:03.158611 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:03.158613 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:03.159427 (+ 814us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:03.159845 (+ 418us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:03.160349 (+ 504us) server_negotiation.cc:299] Negotiation successful
0806 09:27:03.160985 (+ 636us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":791,"thread_start_us":489,"threads_started":1}
I20260806 09:27:03.170197 11813 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec448410f3946b6a7a915adb296ad1e" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 40301 } }
I20260806 09:27:03.170667 11813 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:03.170794 11813 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ec448410f3946b6a7a915adb296ad1e, State: Initialized, Role: FOLLOWER
I20260806 09:27:03.171514 11813 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec448410f3946b6a7a915adb296ad1e" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 40301 } }
I20260806 09:27:03.171833 11813 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:03.171989 11813 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:03.172220 11813 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:03.174798 11813 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec448410f3946b6a7a915adb296ad1e" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 40301 } }
I20260806 09:27:03.175395 11813 leader_election.cc:302] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ec448410f3946b6a7a915adb296ad1e; no voters:
I20260806 09:27:03.176187 11813 leader_election.cc:288] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:03.176494 11820 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:03.178368 11820 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [term 1 LEADER]: Becoming Leader. State: Replica: 7ec448410f3946b6a7a915adb296ad1e, State: Running, Role: LEADER
I20260806 09:27:03.178715 11813 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:03.179234 11820 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec448410f3946b6a7a915adb296ad1e" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 40301 } }
I20260806 09:27:03.183254 11821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ec448410f3946b6a7a915adb296ad1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec448410f3946b6a7a915adb296ad1e" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 40301 } } }
I20260806 09:27:03.183897 11821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:03.184401 11822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ec448410f3946b6a7a915adb296ad1e. Latest consensus state: current_term: 1 leader_uuid: "7ec448410f3946b6a7a915adb296ad1e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ec448410f3946b6a7a915adb296ad1e" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 40301 } } }
I20260806 09:27:03.184804 11822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:03.185824 11831 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:03.198915 11831 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:03.210456 11831 catalog_manager.cc:1347] Generated new cluster ID: 71842cb06af94faba7678ba764b1ccf8
I20260806 09:27:03.210625 11831 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:03.233422 11831 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:03.235140 11831 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:03.248922 11831 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e: Generated new TSK 0
I20260806 09:27:03.249993 11831 catalog_manager.cc:1514] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.16.86.129@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.129@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
Entry for principal kudu/127.16.86.129 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab.
I20260806 09:27:03.341979 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:40301
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:03.750499 11845 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:03.750860 11845 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:03.751019 11845 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:03.758940 11845 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:03.759222 11845 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:03.759442 11845 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:27:03.771252 11845 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.129.keytab
--principal=kudu/127.16.86.129
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:03.772828 11845 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:03.774380 11845 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:03.792662 11850 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:03.793471 11851 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:03.793503 11855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:03.794266 11845 server_base.cc:1034] running on GCE node
I20260806 09:27:03.795737 11845 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:03.797307 11845 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:03.798635 11845 hybrid_clock.cc:648] HybridClock initialized: now 1786008423798540 us; error 85 us; skew 500 ppm
I20260806 09:27:03.799256 11845 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:03 dist-test-slave-h3ft krb5kdc[11703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008423, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:03.803673 11845 init.cc:377] Logged in from keytab as kudu/127.16.86.129@KRBTEST.COM (short username kudu)
I20260806 09:27:03.805850 11845 webserver.cc:460] Webserver started at http://127.16.86.129:33253/ using document root <none> and password file <none>
I20260806 09:27:03.806694 11845 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:03.806869 11845 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:03.807304 11845 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:03.811784 11845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "3301c4d956ed403d95050b50ada76a88"
format_stamp: "Formatted at 2026-08-06 09:27:03 on dist-test-slave-h3ft"
server_key: "5e0d62d3644cf3a7accca230b591ffc3"
server_key_iv: "6863ffb87ae24e77d49c036ebdcd6cfa"
server_key_version: "encryptionkey@0"
I20260806 09:27:03.812875 11845 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "3301c4d956ed403d95050b50ada76a88"
format_stamp: "Formatted at 2026-08-06 09:27:03 on dist-test-slave-h3ft"
server_key: "5e0d62d3644cf3a7accca230b591ffc3"
server_key_iv: "6863ffb87ae24e77d49c036ebdcd6cfa"
server_key_version: "encryptionkey@0"
I20260806 09:27:03.819319 11845 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 09:27:03.824041 11864 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:03.826423 11845 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:27:03.826799 11845 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "3301c4d956ed403d95050b50ada76a88"
format_stamp: "Formatted at 2026-08-06 09:27:03 on dist-test-slave-h3ft"
server_key: "5e0d62d3644cf3a7accca230b591ffc3"
server_key_iv: "6863ffb87ae24e77d49c036ebdcd6cfa"
server_key_version: "encryptionkey@0"
I20260806 09:27:03.827116 11845 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:03.861814 11845 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:03.867599 11845 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:03.868233 11845 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:03.870047 11845 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:27:03.872004 11845 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:27:03.872138 11845 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:03.872332 11845 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:27:03.872414 11845 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:03.917846 11845 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:35043
I20260806 09:27:03.917932 12019 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:35043 every 8 connection(s)
I20260806 09:27:03.919516 11845 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:27:03.924261 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 11845
I20260806 09:27:03.924551 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:27:03.925303 16730 external_mini_cluster.cc:1428] Setting key 742748f94e66d98d86e6881a9fbbd5e9
Aug 06 09:27:03 dist-test-slave-h3ft krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008423, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.129@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:27:03.950500 12024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:03.923436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:57201 (local address 127.16.86.190:40301)
0806 09:27:03.924176 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:03.924184 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:03.925652 (+ 1468us) server_negotiation.cc:407] Connection header received
0806 09:27:03.928898 (+ 3246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:03.928903 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:03.928983 (+ 80us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:03.929136 (+ 153us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:03.932323 (+ 3187us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:03.933154 (+ 831us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:03.936261 (+ 3107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:03.936780 (+ 519us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:03.936923 (+ 143us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:03.943254 (+ 6331us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:03.943262 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:03.943266 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:03.943316 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:03.946440 (+ 3124us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:03.947573 (+ 1133us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:03.947579 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:03.947581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:03.947725 (+ 144us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:03.948249 (+ 524us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:03.948259 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:03.948262 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:03.948610 (+ 348us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:03.948797 (+ 187us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:03.949737 (+ 940us) server_negotiation.cc:299] Negotiation successful
0806 09:27:03.950041 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":362,"threads_started":1}
I20260806 09:27:03.951910 12025 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:03.924200 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:40301 (local address 127.16.86.129:57201)
0806 09:27:03.925348 (+ 1148us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:03.925431 (+ 83us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:03.928109 (+ 2678us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:03.929590 (+ 1481us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:03.929609 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:03.930764 (+ 1155us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:03.932042 (+ 1278us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:03.932078 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:03.933408 (+ 1330us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:03.933416 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:03.935996 (+ 2580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:03.936025 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:03.937129 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:03.937140 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:03.937599 (+ 459us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:03.938635 (+ 1036us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:27:03.938676 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:03.942767 (+ 4091us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:03.946669 (+ 3902us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:03.946682 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:03.946713 (+ 31us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:03.947407 (+ 694us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:03.947856 (+ 449us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:03.947862 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:03.947866 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:03.948108 (+ 242us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:03.948762 (+ 654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:03.948772 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:03.949252 (+ 480us) client_negotiation.cc:769] Sending connection context
0806 09:27:03.949593 (+ 341us) client_negotiation.cc:240] Negotiation successful
0806 09:27:03.950340 (+ 747us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":735,"thread_start_us":411,"threads_started":1}
I20260806 09:27:03.953930 12021 heartbeater.cc:347] Connected to a master server at 127.16.86.190:40301
I20260806 09:27:03.954627 12021 heartbeater.cc:464] Registering TS with master...
I20260806 09:27:03.955783 12021 heartbeater.cc:511] Master 127.16.86.190:40301 requested a full tablet report, sending...
I20260806 09:27:03.959056 11763 ts_manager.cc:194] Registered new tserver with Master: 3301c4d956ed403d95050b50ada76a88 (127.16.86.129:35043)
I20260806 09:27:03.960965 11763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.129@KRBTEST.COM'} at 127.16.86.129:57201
WARNING: no policy specified for kudu/127.16.86.130@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.130@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
Entry for principal kudu/127.16.86.130 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab.
I20260806 09:27:04.021044 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:40301
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:04.415468 12028 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:04.415840 12028 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:04.416042 12028 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:04.424180 12028 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:04.424348 12028 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:04.424543 12028 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:27:04.435731 12028 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.130.keytab
--principal=kudu/127.16.86.130
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:04.437180 12028 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:04.438588 12028 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:04.454048 12035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:04.454763 12038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:04.454710 12036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:04.456497 12028 server_base.cc:1034] running on GCE node
I20260806 09:27:04.457315 12028 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:04.458523 12028 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:04.459997 12028 hybrid_clock.cc:648] HybridClock initialized: now 1786008424459908 us; error 71 us; skew 500 ppm
I20260806 09:27:04.460433 12028 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:04 dist-test-slave-h3ft krb5kdc[11703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008424, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:04.473155 12028 init.cc:377] Logged in from keytab as kudu/127.16.86.130@KRBTEST.COM (short username kudu)
I20260806 09:27:04.475575 12028 webserver.cc:460] Webserver started at http://127.16.86.130:44401/ using document root <none> and password file <none>
I20260806 09:27:04.476912 12028 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:04.477092 12028 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:04.477502 12028 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:04.481118 12028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "bbc4c4d6d0014190a97ddf92789e0e7b"
format_stamp: "Formatted at 2026-08-06 09:27:04 on dist-test-slave-h3ft"
server_key: "b4f3fce72a66e05894a04b19532ea3c9"
server_key_iv: "61036ee31ea21fc7f794c98d32167c4c"
server_key_version: "encryptionkey@0"
I20260806 09:27:04.481987 12028 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "bbc4c4d6d0014190a97ddf92789e0e7b"
format_stamp: "Formatted at 2026-08-06 09:27:04 on dist-test-slave-h3ft"
server_key: "b4f3fce72a66e05894a04b19532ea3c9"
server_key_iv: "61036ee31ea21fc7f794c98d32167c4c"
server_key_version: "encryptionkey@0"
I20260806 09:27:04.488229 12028 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:27:04.492807 12048 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:04.494583 12028 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260806 09:27:04.494875 12028 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "bbc4c4d6d0014190a97ddf92789e0e7b"
format_stamp: "Formatted at 2026-08-06 09:27:04 on dist-test-slave-h3ft"
server_key: "b4f3fce72a66e05894a04b19532ea3c9"
server_key_iv: "61036ee31ea21fc7f794c98d32167c4c"
server_key_version: "encryptionkey@0"
I20260806 09:27:04.495151 12028 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:04.520848 12028 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:04.526041 12028 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:04.526336 12028 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:04.527556 12028 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:27:04.529109 12028 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:27:04.529224 12028 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:04.529384 12028 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:27:04.529461 12028 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:04.571071 12028 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:45583
I20260806 09:27:04.571254 12201 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:45583 every 8 connection(s)
I20260806 09:27:04.572624 12028 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:27:04.575300 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 12028
I20260806 09:27:04.575505 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:27:04.576092 16730 external_mini_cluster.cc:1428] Setting key 9ed9d6cd004cca72be8a6133790489e3
Aug 06 09:27:04 dist-test-slave-h3ft krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008424, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.130@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:27:04.601459 12204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:04.576750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:59973 (local address 127.16.86.190:40301)
0806 09:27:04.577675 (+ 925us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:04.577682 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:04.579072 (+ 1390us) server_negotiation.cc:407] Connection header received
0806 09:27:04.581281 (+ 2209us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:04.581287 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:04.581384 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:04.581524 (+ 140us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:04.583564 (+ 2040us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:04.584837 (+ 1273us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:04.587065 (+ 2228us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:04.587899 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:04.588067 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:04.591958 (+ 3891us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:04.591968 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:04.591976 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:04.592043 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:04.596703 (+ 4660us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:04.598250 (+ 1547us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:04.598255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:04.598258 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:04.598467 (+ 209us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:04.599151 (+ 684us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:04.599156 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:04.599158 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:04.599570 (+ 412us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:04.599709 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:04.600856 (+ 1147us) server_negotiation.cc:299] Negotiation successful
0806 09:27:04.601125 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":769,"thread_start_us":635,"threads_started":1}
I20260806 09:27:04.603096 12205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:04.577637 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:40301 (local address 127.16.86.130:59973)
0806 09:27:04.578845 (+ 1208us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:04.578921 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:04.580956 (+ 2035us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:04.581953 (+ 997us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:04.581963 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:04.582585 (+ 622us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:04.583286 (+ 701us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:04.583306 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:04.585115 (+ 1809us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:04.585119 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:04.586839 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:04.586855 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:04.588190 (+ 1335us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:04.588199 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:04.588425 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:04.589252 (+ 827us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:27:04.589277 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:04.591653 (+ 2376us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:04.597116 (+ 5463us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:04.597134 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:04.597164 (+ 30us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:04.598030 (+ 866us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:04.598622 (+ 592us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:04.598628 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:04.598631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:04.598954 (+ 323us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:04.599716 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:04.599728 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:04.600298 (+ 570us) client_negotiation.cc:769] Sending connection context
0806 09:27:04.600712 (+ 414us) client_negotiation.cc:240] Negotiation successful
0806 09:27:04.601381 (+ 669us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":798,"thread_start_us":452,"threads_started":1}
I20260806 09:27:04.605111 12202 heartbeater.cc:347] Connected to a master server at 127.16.86.190:40301
I20260806 09:27:04.605535 12202 heartbeater.cc:464] Registering TS with master...
I20260806 09:27:04.606519 12202 heartbeater.cc:511] Master 127.16.86.190:40301 requested a full tablet report, sending...
I20260806 09:27:04.608634 11763 ts_manager.cc:194] Registered new tserver with Master: bbc4c4d6d0014190a97ddf92789e0e7b (127.16.86.130:45583)
I20260806 09:27:04.609615 11763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.130@KRBTEST.COM'} at 127.16.86.130:59973
WARNING: no policy specified for kudu/127.16.86.131@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.131@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
Entry for principal kudu/127.16.86.131 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab.
I20260806 09:27:04.673434 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:40301
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:44071
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:27:04.966042 12021 heartbeater.cc:503] Master 127.16.86.190:40301 was elected leader, sending a full tablet report...
W20260806 09:27:05.104744 12209 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:05.105247 12209 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:05.105461 12209 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:05.114799 12209 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:05.115063 12209 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:05.115278 12209 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:27:05.126977 12209 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44071
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.131.keytab
--principal=kudu/127.16.86.131
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:40301
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:05.128870 12209 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:05.130607 12209 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:05.147250 12220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:05.147188 12216 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:05.147730 12217 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:05.149911 12209 server_base.cc:1034] running on GCE node
I20260806 09:27:05.150738 12209 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:05.152009 12209 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:05.153263 12209 hybrid_clock.cc:648] HybridClock initialized: now 1786008425153171 us; error 89 us; skew 500 ppm
I20260806 09:27:05.153676 12209 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:05 dist-test-slave-h3ft krb5kdc[11703](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008425, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:05.157783 12209 init.cc:377] Logged in from keytab as kudu/127.16.86.131@KRBTEST.COM (short username kudu)
I20260806 09:27:05.159699 12209 webserver.cc:460] Webserver started at http://127.16.86.131:41583/ using document root <none> and password file <none>
I20260806 09:27:05.160735 12209 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:05.160893 12209 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:05.161325 12209 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:05.165002 12209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "d752767c42694cc1b08a5c7db023b0c1"
format_stamp: "Formatted at 2026-08-06 09:27:05 on dist-test-slave-h3ft"
server_key: "776f7a5ea41b6cb67b3ab12ffefe6c23"
server_key_iv: "1c9914b6a5ac3cc74d0024b8b7a5a594"
server_key_version: "encryptionkey@0"
I20260806 09:27:05.165907 12209 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "d752767c42694cc1b08a5c7db023b0c1"
format_stamp: "Formatted at 2026-08-06 09:27:05 on dist-test-slave-h3ft"
server_key: "776f7a5ea41b6cb67b3ab12ffefe6c23"
server_key_iv: "1c9914b6a5ac3cc74d0024b8b7a5a594"
server_key_version: "encryptionkey@0"
I20260806 09:27:05.172767 12209 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:27:05.177588 12228 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:05.179876 12209 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:27:05.180156 12209 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "d752767c42694cc1b08a5c7db023b0c1"
format_stamp: "Formatted at 2026-08-06 09:27:05 on dist-test-slave-h3ft"
server_key: "776f7a5ea41b6cb67b3ab12ffefe6c23"
server_key_iv: "1c9914b6a5ac3cc74d0024b8b7a5a594"
server_key_version: "encryptionkey@0"
I20260806 09:27:05.180464 12209 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:05.196853 12209 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:05.203569 12209 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:05.203965 12209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:05.205214 12209 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:27:05.206907 12209 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:27:05.206993 12209 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:05.207186 12209 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:27:05.207242 12209 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:05.252878 12209 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:34435
I20260806 09:27:05.253086 12377 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:34435 every 8 connection(s)
I20260806 09:27:05.254704 12209 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:27:05.260142 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 12209
I20260806 09:27:05.260396 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:27:05.260871 16730 external_mini_cluster.cc:1428] Setting key 5d4550748e31469c51109b05d4d44609
Aug 06 09:27:05 dist-test-slave-h3ft krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008425, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.131@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:27:05.283977 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.258522 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:56261 (local address 127.16.86.190:40301)
0806 09:27:05.259249 (+ 727us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:05.259259 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:05.260095 (+ 836us) server_negotiation.cc:407] Connection header received
0806 09:27:05.262616 (+ 2521us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:05.262623 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:05.262750 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:05.262901 (+ 151us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:05.265091 (+ 2190us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.266056 (+ 965us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.268173 (+ 2117us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.268688 (+ 515us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.268865 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.274686 (+ 5821us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:05.274693 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:05.274698 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:05.274751 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:05.279622 (+ 4871us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:05.280813 (+ 1191us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:05.280820 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:05.280824 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:05.280992 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:05.281537 (+ 545us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:05.281543 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:05.281546 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:05.281920 (+ 374us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:05.282147 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:05.283157 (+ 1010us) server_negotiation.cc:299] Negotiation successful
0806 09:27:05.283475 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":308,"threads_started":1}
I20260806 09:27:05.285450 12383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.258654 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:40301 (local address 127.16.86.131:56261)
0806 09:27:05.259835 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:05.259911 (+ 76us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:05.262255 (+ 2344us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:05.263272 (+ 1017us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:05.263286 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:05.264051 (+ 765us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:05.264861 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:05.264884 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.266262 (+ 1378us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.266269 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:05.267986 (+ 1717us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:05.268002 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.269008 (+ 1006us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.269017 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:05.269317 (+ 300us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.270218 (+ 901us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:27:05.270258 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:05.274250 (+ 3992us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:05.279859 (+ 5609us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:05.279873 (+ 14us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:05.279896 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:05.280643 (+ 747us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:05.281129 (+ 486us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:05.281135 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:05.281139 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:05.281387 (+ 248us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:05.282089 (+ 702us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:05.282101 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:05.282620 (+ 519us) client_negotiation.cc:769] Sending connection context
0806 09:27:05.282974 (+ 354us) client_negotiation.cc:240] Negotiation successful
0806 09:27:05.283820 (+ 846us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":831,"thread_start_us":435,"threads_started":1}
I20260806 09:27:05.287637 12379 heartbeater.cc:347] Connected to a master server at 127.16.86.190:40301
I20260806 09:27:05.288112 12379 heartbeater.cc:464] Registering TS with master...
I20260806 09:27:05.289075 12379 heartbeater.cc:511] Master 127.16.86.190:40301 requested a full tablet report, sending...
I20260806 09:27:05.291253 11763 ts_manager.cc:194] Registered new tserver with Master: d752767c42694cc1b08a5c7db023b0c1 (127.16.86.131:34435)
I20260806 09:27:05.292234 11763 master_service.cc:496] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.16.86.131@KRBTEST.COM'} at 127.16.86.131:56261
I20260806 09:27:05.292428 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:27:05.314600 12382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.298413 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58990 (local address 127.16.86.190:40301)
0806 09:27:05.298715 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:05.298720 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:05.299105 (+ 385us) server_negotiation.cc:407] Connection header received
0806 09:27:05.299386 (+ 281us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:05.299392 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:05.299509 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:05.299767 (+ 258us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:05.301270 (+ 1503us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.302171 (+ 901us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.303804 (+ 1633us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.304365 (+ 561us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.304512 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.307405 (+ 2893us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:05.307414 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:05.307418 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:05.307471 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:05.310937 (+ 3466us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:05.311909 (+ 972us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:05.311915 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:05.311917 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:05.312132 (+ 215us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:05.312808 (+ 676us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:05.312812 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:05.312814 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:05.313291 (+ 477us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:05.313532 (+ 241us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:05.314057 (+ 525us) server_negotiation.cc:299] Negotiation successful
0806 09:27:05.314319 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":119}
I20260806 09:27:05.320026 11763 catalog_manager.cc:1908] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:27:05.328770 11763 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:27:05.360757 12403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.349030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:45583 (local address 127.0.0.1:49248)
0806 09:27:05.350464 (+ 1434us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:05.350484 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:05.350703 (+ 219us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:05.351790 (+ 1087us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:05.351794 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:05.351822 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:27:05.352404 (+ 582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:05.352419 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.354457 (+ 2038us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.354462 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:05.356971 (+ 2509us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:05.356985 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.359423 (+ 2438us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.359431 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:05.359664 (+ 233us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.359734 (+ 70us) client_negotiation.cc:769] Sending connection context
0806 09:27:05.359887 (+ 153us) client_negotiation.cc:240] Negotiation successful
0806 09:27:05.360194 (+ 307us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1310,"mutex_wait_us":59,"thread_start_us":2064,"threads_started":1}
I20260806 09:27:05.361132 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.347113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49248 (local address 127.16.86.130:45583)
0806 09:27:05.348466 (+ 1353us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:05.348478 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:05.350592 (+ 2114us) server_negotiation.cc:407] Connection header received
0806 09:27:05.350921 (+ 329us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:05.350930 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:05.351248 (+ 318us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:05.351471 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:27:05.352645 (+ 1174us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.354279 (+ 1634us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.357149 (+ 2870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.358795 (+ 1646us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.358946 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.359243 (+ 297us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:05.360503 (+ 1260us) server_negotiation.cc:299] Negotiation successful
0806 09:27:05.360738 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1075,"thread_start_us":871,"threads_started":1}
I20260806 09:27:05.361275 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.347521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35446 (local address 127.16.86.131:34435)
0806 09:27:05.348942 (+ 1421us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:05.348955 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:05.350051 (+ 1096us) server_negotiation.cc:407] Connection header received
0806 09:27:05.350527 (+ 476us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:05.350539 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:05.350903 (+ 364us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:05.351163 (+ 260us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:27:05.352432 (+ 1269us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.355270 (+ 2838us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.358350 (+ 3080us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.359969 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.360129 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.360414 (+ 285us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:05.360712 (+ 298us) server_negotiation.cc:299] Negotiation successful
0806 09:27:05.360882 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1162,"thread_start_us":914,"threads_started":1}
I20260806 09:27:05.361857 12400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.347096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:34435 (local address 127.0.0.1:35446)
0806 09:27:05.349923 (+ 2827us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:05.349947 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:05.350241 (+ 294us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:05.351516 (+ 1275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:05.351524 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:05.351559 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:27:05.352078 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:05.352092 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.355669 (+ 3577us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.355679 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:05.358142 (+ 2463us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:05.358154 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.360214 (+ 2060us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.360223 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:05.360507 (+ 284us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.360536 (+ 29us) client_negotiation.cc:769] Sending connection context
0806 09:27:05.360662 (+ 126us) client_negotiation.cc:240] Negotiation successful
0806 09:27:05.360837 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2645,"mutex_wait_us":91,"thread_start_us":2231,"threads_started":1}
I20260806 09:27:05.362587 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.346989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51936 (local address 127.16.86.129:35043)
0806 09:27:05.352033 (+ 5044us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:05.352039 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:05.352063 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:27:05.352152 (+ 89us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:05.352157 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:05.352321 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:05.352564 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:27:05.353164 (+ 600us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.354267 (+ 1103us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.357967 (+ 3700us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.359586 (+ 1619us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.359739 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.360075 (+ 336us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:05.361978 (+ 1903us) server_negotiation.cc:299] Negotiation successful
0806 09:27:05.362181 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4839,"thread_start_us":4686,"threads_started":1}
I20260806 09:27:05.363152 12401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.346484 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35043 (local address 127.0.0.1:51936)
0806 09:27:05.349524 (+ 3040us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:05.349576 (+ 52us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:05.349790 (+ 214us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:05.352534 (+ 2744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:05.352540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:05.352588 (+ 48us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:27:05.352982 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:05.352993 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.354434 (+ 1441us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.354442 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:05.357617 (+ 3175us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:05.357717 (+ 100us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.360265 (+ 2548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.360271 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:05.361528 (+ 1257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.361567 (+ 39us) client_negotiation.cc:769] Sending connection context
0806 09:27:05.361669 (+ 102us) client_negotiation.cc:240] Negotiation successful
0806 09:27:05.361826 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2660,"mutex_wait_us":1096,"thread_start_us":2772,"threads_started":1}
I20260806 09:27:05.366124 12292 tablet_service.cc:1467] Processing CreateTablet for tablet 02afbd97032542a29d5e88da1e4435b5 (DEFAULT_TABLE table=test-table [id=861d076247cf4153b4df5de30336ea0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:27:05.366114 12113 tablet_service.cc:1467] Processing CreateTablet for tablet 02afbd97032542a29d5e88da1e4435b5 (DEFAULT_TABLE table=test-table [id=861d076247cf4153b4df5de30336ea0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:27:05.367386 11939 tablet_service.cc:1467] Processing CreateTablet for tablet 02afbd97032542a29d5e88da1e4435b5 (DEFAULT_TABLE table=test-table [id=861d076247cf4153b4df5de30336ea0a]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:27:05.367933 12113 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02afbd97032542a29d5e88da1e4435b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:05.367931 12292 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02afbd97032542a29d5e88da1e4435b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:05.368873 11939 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 02afbd97032542a29d5e88da1e4435b5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:05.383540 12405 tablet_bootstrap.cc:492] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88: Bootstrap starting.
I20260806 09:27:05.388280 12405 tablet_bootstrap.cc:654] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:05.389873 12405 log.cc:824] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:05.391611 12406 tablet_bootstrap.cc:492] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1: Bootstrap starting.
I20260806 09:27:05.393815 12408 tablet_bootstrap.cc:492] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b: Bootstrap starting.
I20260806 09:27:05.394097 12405 tablet_bootstrap.cc:492] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88: No bootstrap required, opened a new log
I20260806 09:27:05.394461 12405 ts_tablet_manager.cc:1397] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260806 09:27:05.395977 12406 tablet_bootstrap.cc:654] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:05.397107 12406 log.cc:824] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:05.400701 12408 tablet_bootstrap.cc:654] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:05.402341 12408 log.cc:824] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:05.404484 12406 tablet_bootstrap.cc:492] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1: No bootstrap required, opened a new log
I20260806 09:27:05.404953 12406 ts_tablet_manager.cc:1397] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1: Time spent bootstrapping tablet: real 0.014s user 0.005s sys 0.006s
I20260806 09:27:05.405911 12405 raft_consensus.cc:348] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3301c4d956ed403d95050b50ada76a88" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35043 } } peers { permanent_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45583 } } peers { permanent_uuid: "d752767c42694cc1b08a5c7db023b0c1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34435 } }
I20260806 09:27:05.406489 12408 tablet_bootstrap.cc:492] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b: No bootstrap required, opened a new log
I20260806 09:27:05.406495 12405 raft_consensus.cc:374] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:05.406608 12405 raft_consensus.cc:729] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3301c4d956ed403d95050b50ada76a88, State: Initialized, Role: FOLLOWER
I20260806 09:27:05.406837 12408 ts_tablet_manager.cc:1397] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.005s
I20260806 09:27:05.407580 12405 consensus_queue.cc:260] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3301c4d956ed403d95050b50ada76a88" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35043 } } peers { permanent_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45583 } } peers { permanent_uuid: "d752767c42694cc1b08a5c7db023b0c1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34435 } }
I20260806 09:27:05.409816 12405 ts_tablet_manager.cc:1428] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88: Time spent starting tablet: real 0.015s user 0.015s sys 0.000s
I20260806 09:27:05.415162 12406 raft_consensus.cc:348] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3301c4d956ed403d95050b50ada76a88" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35043 } } peers { permanent_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45583 } } peers { permanent_uuid: "d752767c42694cc1b08a5c7db023b0c1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34435 } }
I20260806 09:27:05.416051 12406 raft_consensus.cc:374] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:05.416239 12406 raft_consensus.cc:729] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d752767c42694cc1b08a5c7db023b0c1, State: Initialized, Role: FOLLOWER
I20260806 09:27:05.417127 12406 consensus_queue.cc:260] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3301c4d956ed403d95050b50ada76a88" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35043 } } peers { permanent_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45583 } } peers { permanent_uuid: "d752767c42694cc1b08a5c7db023b0c1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34435 } }
I20260806 09:27:05.417647 12408 raft_consensus.cc:348] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3301c4d956ed403d95050b50ada76a88" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35043 } } peers { permanent_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45583 } } peers { permanent_uuid: "d752767c42694cc1b08a5c7db023b0c1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34435 } }
I20260806 09:27:05.418292 12408 raft_consensus.cc:374] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:05.418429 12408 raft_consensus.cc:729] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bbc4c4d6d0014190a97ddf92789e0e7b, State: Initialized, Role: FOLLOWER
I20260806 09:27:05.418931 12379 heartbeater.cc:503] Master 127.16.86.190:40301 was elected leader, sending a full tablet report...
I20260806 09:27:05.419273 12408 consensus_queue.cc:260] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3301c4d956ed403d95050b50ada76a88" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35043 } } peers { permanent_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45583 } } peers { permanent_uuid: "d752767c42694cc1b08a5c7db023b0c1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34435 } }
I20260806 09:27:05.419669 12406 ts_tablet_manager.cc:1428] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1: Time spent starting tablet: real 0.015s user 0.014s sys 0.002s
I20260806 09:27:05.420557 12202 heartbeater.cc:503] Master 127.16.86.190:40301 was elected leader, sending a full tablet report...
I20260806 09:27:05.421303 12408 ts_tablet_manager.cc:1428] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b: Time spent starting tablet: real 0.014s user 0.011s sys 0.005s
W20260806 09:27:05.423185 12022 tablet.cc:2369] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:27:05.506935 12381 tablet.cc:2369] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260806 09:27:05.575773 12203 tablet.cc:2369] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:27:05.655578 12415 raft_consensus.cc:482] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:27:05.655983 12415 raft_consensus.cc:504] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3301c4d956ed403d95050b50ada76a88" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35043 } } peers { permanent_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45583 } } peers { permanent_uuid: "d752767c42694cc1b08a5c7db023b0c1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34435 } }
I20260806 09:27:05.658324 12415 leader_election.cc:288] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3301c4d956ed403d95050b50ada76a88 (127.16.86.129:35043), d752767c42694cc1b08a5c7db023b0c1 (127.16.86.131:34435)
I20260806 09:27:05.667742 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.659267 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:51547 (local address 127.16.86.131:34435)
0806 09:27:05.659575 (+ 308us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:05.659585 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:05.660634 (+ 1049us) server_negotiation.cc:407] Connection header received
0806 09:27:05.660885 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:05.660890 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:05.660972 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:05.661106 (+ 134us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:27:05.661973 (+ 867us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.662782 (+ 809us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.665521 (+ 2739us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.666362 (+ 841us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.666523 (+ 161us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.666630 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:05.667303 (+ 673us) server_negotiation.cc:299] Negotiation successful
0806 09:27:05.667448 (+ 145us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":103}
I20260806 09:27:05.667825 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.659059 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:43593 (local address 127.16.86.129:35043)
0806 09:27:05.659311 (+ 252us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:05.659319 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:05.660435 (+ 1116us) server_negotiation.cc:407] Connection header received
0806 09:27:05.660842 (+ 407us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:05.660848 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:05.660939 (+ 91us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:05.661025 (+ 86us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:27:05.661780 (+ 755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.662666 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.665383 (+ 2717us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.666492 (+ 1109us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.666733 (+ 241us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.666842 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:05.667259 (+ 417us) server_negotiation.cc:299] Negotiation successful
0806 09:27:05.667437 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 09:27:05.667922 12416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.658886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:35043 (local address 127.16.86.130:43593)
0806 09:27:05.660279 (+ 1393us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:05.660314 (+ 35us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:05.660536 (+ 222us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:05.661089 (+ 553us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:05.661094 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:05.661124 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:27:05.661581 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:05.661597 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.662852 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.662859 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:05.665119 (+ 2260us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:05.665133 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.666815 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.666821 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:05.667035 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.667069 (+ 34us) client_negotiation.cc:769] Sending connection context
0806 09:27:05.667161 (+ 92us) client_negotiation.cc:240] Negotiation successful
0806 09:27:05.667354 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1202,"mutex_wait_us":168,"thread_start_us":690,"threads_started":1}
I20260806 09:27:05.668015 12417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.659059 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:34435 (local address 127.16.86.130:51547)
0806 09:27:05.660465 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:05.660484 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:05.660695 (+ 211us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:05.661164 (+ 469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:05.661169 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:05.661203 (+ 34us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:27:05.661771 (+ 568us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:05.661786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.662990 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.662995 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:05.665253 (+ 2258us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:05.665282 (+ 29us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.666676 (+ 1394us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.666684 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:05.667043 (+ 359us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.667083 (+ 40us) client_negotiation.cc:769] Sending connection context
0806 09:27:05.667241 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 09:27:05.667436 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1280,"mutex_wait_us":11,"thread_start_us":876,"threads_started":1}
I20260806 09:27:05.669478 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02afbd97032542a29d5e88da1e4435b5" candidate_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3301c4d956ed403d95050b50ada76a88" is_pre_election: true
I20260806 09:27:05.669453 12322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02afbd97032542a29d5e88da1e4435b5" candidate_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d752767c42694cc1b08a5c7db023b0c1" is_pre_election: true
I20260806 09:27:05.670229 12322 raft_consensus.cc:2454] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbc4c4d6d0014190a97ddf92789e0e7b in term 0.
I20260806 09:27:05.670229 11964 raft_consensus.cc:2454] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bbc4c4d6d0014190a97ddf92789e0e7b in term 0.
I20260806 09:27:05.671113 12050 leader_election.cc:302] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3301c4d956ed403d95050b50ada76a88, bbc4c4d6d0014190a97ddf92789e0e7b, d752767c42694cc1b08a5c7db023b0c1; no voters:
I20260806 09:27:05.671650 12415 raft_consensus.cc:2781] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:27:05.671955 12415 raft_consensus.cc:482] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:27:05.672120 12415 raft_consensus.cc:3037] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:05.675678 12415 raft_consensus.cc:504] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3301c4d956ed403d95050b50ada76a88" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35043 } } peers { permanent_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45583 } } peers { permanent_uuid: "d752767c42694cc1b08a5c7db023b0c1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34435 } }
I20260806 09:27:05.676704 12415 leader_election.cc:288] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [CANDIDATE]: Term 1 election: Requested vote from peers 3301c4d956ed403d95050b50ada76a88 (127.16.86.129:35043), d752767c42694cc1b08a5c7db023b0c1 (127.16.86.131:34435)
I20260806 09:27:05.677449 11964 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02afbd97032542a29d5e88da1e4435b5" candidate_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3301c4d956ed403d95050b50ada76a88"
I20260806 09:27:05.677618 12322 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "02afbd97032542a29d5e88da1e4435b5" candidate_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d752767c42694cc1b08a5c7db023b0c1"
I20260806 09:27:05.677742 11964 raft_consensus.cc:3037] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:05.677937 12322 raft_consensus.cc:3037] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:05.680542 12322 raft_consensus.cc:2454] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbc4c4d6d0014190a97ddf92789e0e7b in term 1.
I20260806 09:27:05.681061 11964 raft_consensus.cc:2454] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bbc4c4d6d0014190a97ddf92789e0e7b in term 1.
I20260806 09:27:05.681109 12050 leader_election.cc:302] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: bbc4c4d6d0014190a97ddf92789e0e7b, d752767c42694cc1b08a5c7db023b0c1; no voters:
I20260806 09:27:05.681627 12415 raft_consensus.cc:2781] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:05.682299 12415 raft_consensus.cc:686] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 1 LEADER]: Becoming Leader. State: Replica: bbc4c4d6d0014190a97ddf92789e0e7b, State: Running, Role: LEADER
I20260806 09:27:05.682981 12415 consensus_queue.cc:237] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3301c4d956ed403d95050b50ada76a88" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35043 } } peers { permanent_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45583 } } peers { permanent_uuid: "d752767c42694cc1b08a5c7db023b0c1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34435 } }
I20260806 09:27:05.689648 11763 catalog_manager.cc:5495] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b reported cstate change: term changed from 0 to 1, leader changed from <none> to bbc4c4d6d0014190a97ddf92789e0e7b (127.16.86.130). New cstate: current_term: 1 leader_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3301c4d956ed403d95050b50ada76a88" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35043 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "bbc4c4d6d0014190a97ddf92789e0e7b" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 45583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d752767c42694cc1b08a5c7db023b0c1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34435 } health_report { overall_health: UNKNOWN } } }
I20260806 09:27:05.737815 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.730579 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49262 (local address 127.16.86.130:45583)
0806 09:27:05.730833 (+ 254us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:05.730838 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:05.730867 (+ 29us) server_negotiation.cc:407] Connection header received
0806 09:27:05.731113 (+ 246us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:05.731124 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:05.731270 (+ 146us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:05.731483 (+ 213us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:27:05.731980 (+ 497us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.732989 (+ 1009us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.734830 (+ 1841us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.735415 (+ 585us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.735590 (+ 175us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.736381 (+ 791us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:27:05.737139 (+ 758us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:27:05.737292 (+ 153us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:05.737428 (+ 136us) server_negotiation.cc:299] Negotiation successful
0806 09:27:05.737527 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20260806 09:27:05.749428 12322 raft_consensus.cc:1261] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1 [term 1 FOLLOWER]: Refusing update from remote peer bbc4c4d6d0014190a97ddf92789e0e7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:27:05.749632 11964 raft_consensus.cc:1261] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88 [term 1 FOLLOWER]: Refusing update from remote peer bbc4c4d6d0014190a97ddf92789e0e7b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:27:05.750903 12418 consensus_queue.cc:1035] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "d752767c42694cc1b08a5c7db023b0c1" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 34435 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:27:05.751459 12415 consensus_queue.cc:1035] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [LEADER]: Connected to new peer: Peer: permanent_uuid: "3301c4d956ed403d95050b50ada76a88" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 35043 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:27:05.766815 12422 mvcc.cc:205] Tried to move back new op lower bound from 7315490511855759360 to 7315490511603060736. Current Snapshot: MvccSnapshot[applied={T|T < 7315490511855759360}]
I20260806 09:27:05.768980 12423 mvcc.cc:205] Tried to move back new op lower bound from 7315490511855759360 to 7315490511603060736. Current Snapshot: MvccSnapshot[applied={T|T < 7315490511855759360}]
I20260806 09:27:05.783763 11762 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:27:05.784227 11762 catalog_manager.cc:2645] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58990:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:27:05.789327 11762 catalog_manager.cc:5782] T 00000000000000000000000000000000 P 7ec448410f3946b6a7a915adb296ad1e: Sending DeleteTablet for 3 replicas of tablet 02afbd97032542a29d5e88da1e4435b5
I20260806 09:27:05.790933 11939 tablet_service.cc:1514] Processing DeleteTablet for tablet 02afbd97032542a29d5e88da1e4435b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:27:05 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:51936
I20260806 09:27:05.790922 12113 tablet_service.cc:1514] Processing DeleteTablet for tablet 02afbd97032542a29d5e88da1e4435b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:27:05 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:49248
I20260806 09:27:05.791033 12292 tablet_service.cc:1514] Processing DeleteTablet for tablet 02afbd97032542a29d5e88da1e4435b5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:27:05 UTC) from {username='kudu', principal='kudu/127.16.86.190@KRBTEST.COM'} at 127.0.0.1:35446
I20260806 09:27:05.795779 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 11845
I20260806 09:27:05.802084 12437 tablet_replica.cc:330] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88: stopping tablet replica
I20260806 09:27:05.802760 12437 raft_consensus.cc:2229] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:27:05.805261 12437 raft_consensus.cc:2258] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:27:05.809247 12437 ts_tablet_manager.cc:1905] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:27:05.811897 12435 tablet_replica.cc:330] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b: stopping tablet replica
I20260806 09:27:05.812665 12438 tablet_replica.cc:330] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1: stopping tablet replica
I20260806 09:27:05.813362 12438 raft_consensus.cc:2229] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:27:05.814051 12438 raft_consensus.cc:2258] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:27:05.819909 12435 raft_consensus.cc:2229] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 1 LEADER]: Raft consensus shutting down.
Aug 06 09:27:05 dist-test-slave-h3ft krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.129@KRBTEST.COM
I20260806 09:27:05.820863 12435 raft_consensus.cc:2258] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:27:05.823052 12438 ts_tablet_manager.cc:1905] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:27:05.823805 12435 ts_tablet_manager.cc:1905] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:27:05.824340 12437 ts_tablet_manager.cc:1918] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:27:05.824775 12437 log.cc:1196] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/02afbd97032542a29d5e88da1e4435b5
I20260806 09:27:05.825603 12437 ts_tablet_manager.cc:1939] T 02afbd97032542a29d5e88da1e4435b5 P 3301c4d956ed403d95050b50ada76a88: Deleting consensus metadata
I20260806 09:27:05.828649 11739 catalog_manager.cc:4841] TS 3301c4d956ed403d95050b50ada76a88 (127.16.86.129:35043): tablet 02afbd97032542a29d5e88da1e4435b5 (table test-table [id=861d076247cf4153b4df5de30336ea0a]) successfully deleted
I20260806 09:27:05.831863 12438 ts_tablet_manager.cc:1918] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:27:05.832662 12438 log.cc:1196] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/02afbd97032542a29d5e88da1e4435b5
I20260806 09:27:05.833488 12438 ts_tablet_manager.cc:1939] T 02afbd97032542a29d5e88da1e4435b5 P d752767c42694cc1b08a5c7db023b0c1: Deleting consensus metadata
I20260806 09:27:05.834576 12396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.797346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51938 (local address 127.16.86.129:35043)
0806 09:27:05.797607 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:05.797616 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:05.809401 (+ 11785us) server_negotiation.cc:407] Connection header received
0806 09:27:05.810221 (+ 820us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:05.810233 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:05.810397 (+ 164us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:05.810513 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:05.813004 (+ 2491us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.813855 (+ 851us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.815639 (+ 1784us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.816116 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.816297 (+ 181us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.824531 (+ 8234us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:05.824553 (+ 22us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:05.824573 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:05.824624 (+ 51us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:05.829122 (+ 4498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:05.830406 (+ 1284us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:05.830419 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:05.830423 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:05.830712 (+ 289us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:05.831900 (+ 1188us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:05.831905 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:05.831909 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:05.832652 (+ 743us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:05.833049 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:05.833843 (+ 794us) server_negotiation.cc:299] Negotiation successful
0806 09:27:05.834134 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20260806 09:27:05.835891 11740 catalog_manager.cc:4841] TS d752767c42694cc1b08a5c7db023b0c1 (127.16.86.131:34435): tablet 02afbd97032542a29d5e88da1e4435b5 (table test-table [id=861d076247cf4153b4df5de30336ea0a]) successfully deleted
I20260806 09:27:05.836647 12014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:05.838501 12435 ts_tablet_manager.cc:1918] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:27:05.838876 12435 log.cc:1196] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/02afbd97032542a29d5e88da1e4435b5
I20260806 09:27:05.839917 12435 ts_tablet_manager.cc:1939] T 02afbd97032542a29d5e88da1e4435b5 P bbc4c4d6d0014190a97ddf92789e0e7b: Deleting consensus metadata
I20260806 09:27:05.843246 11742 catalog_manager.cc:4841] TS bbc4c4d6d0014190a97ddf92789e0e7b (127.16.86.130:45583): tablet 02afbd97032542a29d5e88da1e4435b5 (table test-table [id=861d076247cf4153b4df5de30336ea0a]) successfully deleted
I20260806 09:27:05.948002 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 11845
I20260806 09:27:05.974244 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 12028
Aug 06 09:27:05 dist-test-slave-h3ft krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.130@KRBTEST.COM
I20260806 09:27:05.997196 12397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:05.975334 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49266 (local address 127.16.86.130:45583)
0806 09:27:05.975693 (+ 359us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:05.975701 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:05.975731 (+ 30us) server_negotiation.cc:407] Connection header received
0806 09:27:05.975978 (+ 247us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:05.975985 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:05.976088 (+ 103us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:05.976230 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:05.977943 (+ 1713us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.979518 (+ 1575us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.981014 (+ 1496us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:05.981754 (+ 740us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:05.981901 (+ 147us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:05.987811 (+ 5910us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:05.987835 (+ 24us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:05.987864 (+ 29us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:05.987925 (+ 61us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:05.993010 (+ 5085us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:05.993928 (+ 918us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:05.993941 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:05.993945 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:05.994170 (+ 225us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:05.994639 (+ 469us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:05.994644 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:05.994648 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:05.995268 (+ 620us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:05.995685 (+ 417us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:05.996108 (+ 423us) server_negotiation.cc:299] Negotiation successful
0806 09:27:05.996426 (+ 318us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":138}
I20260806 09:27:05.998436 12196 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:06.132905 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 12028
I20260806 09:27:06.164031 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 12209
Aug 06 09:27:06 dist-test-slave-h3ft krb5kdc[11703](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008422, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.131@KRBTEST.COM
I20260806 09:27:06.186343 12402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:06.165095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35456 (local address 127.16.86.131:34435)
0806 09:27:06.165405 (+ 310us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:06.165411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:06.165445 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:27:06.165527 (+ 82us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:06.165532 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:06.165617 (+ 85us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:06.165725 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:06.167482 (+ 1757us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:06.168413 (+ 931us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:06.170037 (+ 1624us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:06.170535 (+ 498us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:06.170671 (+ 136us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:06.177718 (+ 7047us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:06.177745 (+ 27us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:06.177769 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:06.177835 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:06.182066 (+ 4231us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:06.183236 (+ 1170us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:06.183246 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:06.183249 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:06.183524 (+ 275us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:06.184104 (+ 580us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:06.184110 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:06.184116 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:06.184755 (+ 639us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:06.185201 (+ 446us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:06.185554 (+ 353us) server_negotiation.cc:299] Negotiation successful
0806 09:27:06.185854 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":131}
I20260806 09:27:06.187358 12373 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:06.293111 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 12209
I20260806 09:27:06.327224 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 11719
I20260806 09:27:06.328703 11807 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:06.465646 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 11719
2026-08-06T09:27:06Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5755 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2026-08-06T09:27:06Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:06Z Disabled control of system clock
I20260806 09:27:06.596998 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:38383
--webserver_interface=127.16.86.190
--webserver_port=0
--builtin_ntp_servers=127.16.86.148:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:38383
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:27:07.035269 12453 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:07.035706 12453 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:07.035806 12453 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:07.044667 12453 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:07.044829 12453 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:07.044875 12453 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:07.044975 12453 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:07.045049 12453 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:07.062227 12453 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:38383
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:38383
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:07.064860 12453 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:07.068490 12453 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:07.087724 12463 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:07.087867 12460 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:07.088505 12459 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:07.093037 12453 server_base.cc:1034] running on GCE node
I20260806 09:27:07.094761 12453 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:07.097728 12453 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:07.101333 12453 hybrid_clock.cc:648] HybridClock initialized: now 1786008427101139 us; error 135 us; skew 500 ppm
I20260806 09:27:07.102133 12453 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:27:07.106791 12453 webserver.cc:460] Webserver started at http://127.16.86.190:45775/ using document root <none> and password file <none>
I20260806 09:27:07.108270 12453 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:07.108464 12453 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:07.109019 12453 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:07.114614 12453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "a2f734d5f35a44cd9ffffa58a0ea4dfb"
format_stamp: "Formatted at 2026-08-06 09:27:07 on dist-test-slave-h3ft"
server_key: "3d451ea68ba528e0a4a1bed1a0bfe74b"
server_key_iv: "b652a9f3b945d3adc82a7f35aff9c9d2"
server_key_version: "encryptionkey@0"
I20260806 09:27:07.115767 12453 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "a2f734d5f35a44cd9ffffa58a0ea4dfb"
format_stamp: "Formatted at 2026-08-06 09:27:07 on dist-test-slave-h3ft"
server_key: "3d451ea68ba528e0a4a1bed1a0bfe74b"
server_key_iv: "b652a9f3b945d3adc82a7f35aff9c9d2"
server_key_version: "encryptionkey@0"
I20260806 09:27:07.122865 12453 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260806 09:27:07.127751 12473 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:07.129773 12453 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260806 09:27:07.130079 12453 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "a2f734d5f35a44cd9ffffa58a0ea4dfb"
format_stamp: "Formatted at 2026-08-06 09:27:07 on dist-test-slave-h3ft"
server_key: "3d451ea68ba528e0a4a1bed1a0bfe74b"
server_key_iv: "b652a9f3b945d3adc82a7f35aff9c9d2"
server_key_version: "encryptionkey@0"
I20260806 09:27:07.130400 12453 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:07.146061 12453 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:07.147279 12453 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:07.147635 12453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:07.172330 12453 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:38383
I20260806 09:27:07.172363 12541 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:38383 every 8 connection(s)
I20260806 09:27:07.174230 12453 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:07.178808 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 12453
I20260806 09:27:07.179051 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:07.179231 12542 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:07.179710 16730 external_mini_cluster.cc:1428] Setting key 176f348ca18f02ca8e8b94fb8a95cd61
I20260806 09:27:07.195389 12542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb: Bootstrap starting.
I20260806 09:27:07.198607 12546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:07.182388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50744 (local address 127.16.86.190:38383)
0806 09:27:07.184010 (+ 1622us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:07.184025 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:07.184085 (+ 60us) server_negotiation.cc:407] Connection header received
0806 09:27:07.186737 (+ 2652us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:07.186770 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:07.187591 (+ 821us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:07.188093 (+ 502us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:07.188750 (+ 657us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:07.190446 (+ 1696us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:07.192668 (+ 2222us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:07.193651 (+ 983us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:07.193828 (+ 177us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:07.195311 (+ 1483us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:07.195321 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:07.195340 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:07.196224 (+ 884us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:07.196342 (+ 118us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:07.196370 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:07.196493 (+ 123us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:07.196776 (+ 283us) server_negotiation.cc:299] Negotiation successful
0806 09:27:07.197325 (+ 549us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1024,"thread_start_us":661,"threads_started":1}
I20260806 09:27:07.199555 12542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:07.200752 12542 log.cc:824] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:07.204255 12542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb: No bootstrap required, opened a new log
I20260806 09:27:07.212601 12542 raft_consensus.cc:348] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f734d5f35a44cd9ffffa58a0ea4dfb" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38383 } }
I20260806 09:27:07.213095 12542 raft_consensus.cc:374] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:07.213230 12542 raft_consensus.cc:729] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2f734d5f35a44cd9ffffa58a0ea4dfb, State: Initialized, Role: FOLLOWER
I20260806 09:27:07.214216 12542 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f734d5f35a44cd9ffffa58a0ea4dfb" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38383 } }
I20260806 09:27:07.214803 12542 raft_consensus.cc:388] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:07.214978 12542 raft_consensus.cc:482] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:07.215216 12542 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:07.217756 12542 raft_consensus.cc:504] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f734d5f35a44cd9ffffa58a0ea4dfb" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38383 } }
I20260806 09:27:07.218369 12542 leader_election.cc:302] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2f734d5f35a44cd9ffffa58a0ea4dfb; no voters:
I20260806 09:27:07.219273 12542 leader_election.cc:288] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:07.219586 12548 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:07.220885 12548 raft_consensus.cc:686] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [term 1 LEADER]: Becoming Leader. State: Replica: a2f734d5f35a44cd9ffffa58a0ea4dfb, State: Running, Role: LEADER
I20260806 09:27:07.221836 12542 sys_catalog.cc:564] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:07.221745 12548 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f734d5f35a44cd9ffffa58a0ea4dfb" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38383 } }
I20260806 09:27:07.225630 12550 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2f734d5f35a44cd9ffffa58a0ea4dfb. Latest consensus state: current_term: 1 leader_uuid: "a2f734d5f35a44cd9ffffa58a0ea4dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f734d5f35a44cd9ffffa58a0ea4dfb" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38383 } } }
I20260806 09:27:07.226094 12550 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:07.226539 12549 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2f734d5f35a44cd9ffffa58a0ea4dfb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2f734d5f35a44cd9ffffa58a0ea4dfb" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38383 } } }
I20260806 09:27:07.227236 12549 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:07.229192 12557 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:07.237327 12557 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:07.250133 12557 catalog_manager.cc:1347] Generated new cluster ID: 6f1a8f36453c489ba3ff3628d8ac2792
I20260806 09:27:07.250324 12557 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:07.267941 12557 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:07.269999 12557 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:07.298271 12557 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb: Generated new TSK 0
I20260806 09:27:07.299690 12557 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:07.311529 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.129:0
--local_ip_for_outbound_sockets=127.16.86.129
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38383
--builtin_ntp_servers=127.16.86.148:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20260806 09:27:07.758000 12571 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:07.758409 12571 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:07.758579 12571 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:07.766852 12571 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:07.767035 12571 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:07.767241 12571 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.129
I20260806 09:27:07.777539 12571 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.129:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.16.86.129
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.129
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:07.779417 12571 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:07.781812 12571 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:07.798710 12579 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:07.798916 12585 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:07.799635 12578 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:07.805096 12571 server_base.cc:1034] running on GCE node
I20260806 09:27:07.806020 12571 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:07.807410 12571 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:07.808696 12571 hybrid_clock.cc:648] HybridClock initialized: now 1786008427808639 us; error 44 us; skew 500 ppm
I20260806 09:27:07.809126 12571 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:27:07.812284 12571 webserver.cc:460] Webserver started at http://127.16.86.129:36079/ using document root <none> and password file <none>
I20260806 09:27:07.813251 12571 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:07.813402 12571 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:07.813804 12571 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:07.817584 12571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/data/instance:
uuid: "97dcecd806344effba2d54e49295f6bc"
format_stamp: "Formatted at 2026-08-06 09:27:07 on dist-test-slave-h3ft"
server_key: "9dcaceda8c7d65e8347da139ea70fb29"
server_key_iv: "ed7f811338ab00843a11ff01da766f5a"
server_key_version: "encryptionkey@0"
I20260806 09:27:07.818555 12571 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance:
uuid: "97dcecd806344effba2d54e49295f6bc"
format_stamp: "Formatted at 2026-08-06 09:27:07 on dist-test-slave-h3ft"
server_key: "9dcaceda8c7d65e8347da139ea70fb29"
server_key_iv: "ed7f811338ab00843a11ff01da766f5a"
server_key_version: "encryptionkey@0"
I20260806 09:27:07.825348 12571 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260806 09:27:07.829871 12591 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:07.832010 12571 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:27:07.832249 12571 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/wal
uuid: "97dcecd806344effba2d54e49295f6bc"
format_stamp: "Formatted at 2026-08-06 09:27:07 on dist-test-slave-h3ft"
server_key: "9dcaceda8c7d65e8347da139ea70fb29"
server_key_iv: "ed7f811338ab00843a11ff01da766f5a"
server_key_version: "encryptionkey@0"
I20260806 09:27:07.832581 12571 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:07.873723 12571 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:07.874886 12571 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:07.875319 12571 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:07.876690 12571 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:27:07.878640 12571 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:27:07.878786 12571 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:07.878949 12571 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:27:07.879078 12571 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:07.922860 12571 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.129:33797
I20260806 09:27:07.922973 12751 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.129:33797 every 8 connection(s)
I20260806 09:27:07.924657 12571 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/data/info.pb
I20260806 09:27:07.928179 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 12571
I20260806 09:27:07.928404 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/wal/instance
I20260806 09:27:07.929222 16730 external_mini_cluster.cc:1428] Setting key b7e0e4f0a6574fc21e578b13c05ad103
I20260806 09:27:07.932787 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.130:0
--local_ip_for_outbound_sockets=127.16.86.130
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38383
--builtin_ntp_servers=127.16.86.148:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:27:07.941260 12755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:07.928341 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:42363 (local address 127.16.86.190:38383)
0806 09:27:07.929230 (+ 889us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:07.929238 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:07.930848 (+ 1610us) server_negotiation.cc:407] Connection header received
0806 09:27:07.933046 (+ 2198us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:07.933052 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:07.933164 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:07.933303 (+ 139us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:07.934468 (+ 1165us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:07.935346 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:07.937454 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:07.938225 (+ 771us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:07.938393 (+ 168us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:07.940230 (+ 1837us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:07.940235 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:07.940239 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:07.940259 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:07.940310 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:07.940324 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:07.940433 (+ 109us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:07.940746 (+ 313us) server_negotiation.cc:299] Negotiation successful
0806 09:27:07.940867 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":598,"threads_started":1}
I20260806 09:27:07.942369 12756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:07.929378 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:38383 (local address 127.16.86.129:42363)
0806 09:27:07.930623 (+ 1245us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:07.930681 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:07.932709 (+ 2028us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:07.933559 (+ 850us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:07.933568 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:07.933659 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:07.934289 (+ 630us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:07.934306 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:07.935679 (+ 1373us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:07.935684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:07.937276 (+ 1592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:07.937292 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:07.938523 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:07.938530 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:07.938793 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:07.939825 (+ 1032us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:27:07.939828 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:07.939905 (+ 77us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:27:07.940007 (+ 102us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:07.940461 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:07.940477 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:07.940490 (+ 13us) client_negotiation.cc:769] Sending connection context
0806 09:27:07.940732 (+ 242us) client_negotiation.cc:240] Negotiation successful
0806 09:27:07.941175 (+ 443us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":414,"threads_started":1}
I20260806 09:27:07.945003 12752 heartbeater.cc:347] Connected to a master server at 127.16.86.190:38383
I20260806 09:27:07.945461 12752 heartbeater.cc:464] Registering TS with master...
I20260806 09:27:07.946373 12752 heartbeater.cc:511] Master 127.16.86.190:38383 requested a full tablet report, sending...
I20260806 09:27:07.949805 12495 ts_manager.cc:194] Registered new tserver with Master: 97dcecd806344effba2d54e49295f6bc (127.16.86.129:33797)
I20260806 09:27:07.952672 12495 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.86.129:42363
W20260806 09:27:08.365088 12757 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:08.365466 12757 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:08.365639 12757 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:08.373628 12757 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:08.373795 12757 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:08.374023 12757 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.130
I20260806 09:27:08.384266 12757 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.130:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.16.86.130
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.130
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:08.386147 12757 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:08.388502 12757 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:08.404476 12768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:08.404520 12763 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:08.404376 12762 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:08.406114 12757 server_base.cc:1034] running on GCE node
I20260806 09:27:08.406935 12757 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:08.408147 12757 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:08.409443 12757 hybrid_clock.cc:648] HybridClock initialized: now 1786008428409391 us; error 48 us; skew 500 ppm
I20260806 09:27:08.409789 12757 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:27:08.412441 12757 webserver.cc:460] Webserver started at http://127.16.86.130:37313/ using document root <none> and password file <none>
I20260806 09:27:08.413254 12757 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:08.413352 12757 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:08.413642 12757 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:08.416752 12757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/data/instance:
uuid: "ecfff36343a7447abea024b6aace647a"
format_stamp: "Formatted at 2026-08-06 09:27:08 on dist-test-slave-h3ft"
server_key: "f95faa20c8bf767bde914b648edf0903"
server_key_iv: "5af1c8d8bd6db2a304e457e315e07111"
server_key_version: "encryptionkey@0"
I20260806 09:27:08.417508 12757 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance:
uuid: "ecfff36343a7447abea024b6aace647a"
format_stamp: "Formatted at 2026-08-06 09:27:08 on dist-test-slave-h3ft"
server_key: "f95faa20c8bf767bde914b648edf0903"
server_key_iv: "5af1c8d8bd6db2a304e457e315e07111"
server_key_version: "encryptionkey@0"
I20260806 09:27:08.423780 12757 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.002s sys 0.005s
I20260806 09:27:08.428203 12775 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:08.429884 12757 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:27:08.430078 12757 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/wal
uuid: "ecfff36343a7447abea024b6aace647a"
format_stamp: "Formatted at 2026-08-06 09:27:08 on dist-test-slave-h3ft"
server_key: "f95faa20c8bf767bde914b648edf0903"
server_key_iv: "5af1c8d8bd6db2a304e457e315e07111"
server_key_version: "encryptionkey@0"
I20260806 09:27:08.430308 12757 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:08.444818 12757 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:08.445703 12757 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:08.446040 12757 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:08.447240 12757 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:27:08.448904 12757 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:27:08.449038 12757 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:08.449218 12757 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:27:08.449319 12757 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:08.490856 12757 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.130:42407
I20260806 09:27:08.491043 12928 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.130:42407 every 8 connection(s)
I20260806 09:27:08.492360 12757 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/data/info.pb
I20260806 09:27:08.500489 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 12757
I20260806 09:27:08.500726 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/wal/instance
I20260806 09:27:08.501363 16730 external_mini_cluster.cc:1428] Setting key d375800ae2955c51f4bb614ea4f52329
I20260806 09:27:08.504633 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.16.86.131:0
--local_ip_for_outbound_sockets=127.16.86.131
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38383
--builtin_ntp_servers=127.16.86.148:44405
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20260806 09:27:08.508606 12932 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:08.495784 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:57485 (local address 127.16.86.190:38383)
0806 09:27:08.496353 (+ 569us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:08.496360 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:08.497939 (+ 1579us) server_negotiation.cc:407] Connection header received
0806 09:27:08.500024 (+ 2085us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:08.500031 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:08.500145 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:08.500301 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:08.501930 (+ 1629us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:08.502808 (+ 878us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:08.504916 (+ 2108us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:08.505393 (+ 477us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:08.505553 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:08.507467 (+ 1914us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:08.507473 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:08.507476 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:08.507501 (+ 25us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:08.507552 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:08.507567 (+ 15us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:08.507678 (+ 111us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:08.507979 (+ 301us) server_negotiation.cc:299] Negotiation successful
0806 09:27:08.508107 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":244,"threads_started":1}
I20260806 09:27:08.509838 12933 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:08.496560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:38383 (local address 127.16.86.130:57485)
0806 09:27:08.497740 (+ 1180us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:08.497796 (+ 56us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:08.499604 (+ 1808us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:08.500650 (+ 1046us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:08.500664 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:08.500794 (+ 130us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:08.501664 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:08.501692 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:08.503072 (+ 1380us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:08.503082 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:08.504607 (+ 1525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:08.504626 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:08.505687 (+ 1061us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:08.505696 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:08.505990 (+ 294us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:08.506973 (+ 983us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:27:08.506976 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:08.507093 (+ 117us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:27:08.507200 (+ 107us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:08.507678 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:08.507704 (+ 26us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:08.507721 (+ 17us) client_negotiation.cc:769] Sending connection context
0806 09:27:08.507948 (+ 227us) client_negotiation.cc:240] Negotiation successful
0806 09:27:08.508441 (+ 493us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"thread_start_us":346,"threads_started":1}
I20260806 09:27:08.511760 12930 heartbeater.cc:347] Connected to a master server at 127.16.86.190:38383
I20260806 09:27:08.512169 12930 heartbeater.cc:464] Registering TS with master...
I20260806 09:27:08.513756 12930 heartbeater.cc:511] Master 127.16.86.190:38383 requested a full tablet report, sending...
I20260806 09:27:08.515791 12495 ts_manager.cc:194] Registered new tserver with Master: ecfff36343a7447abea024b6aace647a (127.16.86.130:42407)
I20260806 09:27:08.517092 12495 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.86.130:57485
W20260806 09:27:08.944073 12934 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:08.944494 12934 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:08.944665 12934 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:08.953197 12934 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:08.953359 12934 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:08.953574 12934 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.16.86.131
I20260806 09:27:08.957669 12752 heartbeater.cc:503] Master 127.16.86.190:38383 was elected leader, sending a full tablet report...
I20260806 09:27:08.964949 12934 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:44405
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.16.86.131:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.16.86.131
--webserver_port=0
--tserver_master_addrs=127.16.86.190:38383
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.16.86.131
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:08.966765 12934 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:08.969337 12934 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:08.986918 12941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:08.987033 12942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:08.987089 12944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:08.988116 12934 server_base.cc:1034] running on GCE node
I20260806 09:27:08.988976 12934 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:08.990342 12934 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:08.991655 12934 hybrid_clock.cc:648] HybridClock initialized: now 1786008428991582 us; error 102 us; skew 500 ppm
I20260806 09:27:08.992164 12934 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260806 09:27:08.995260 12934 webserver.cc:460] Webserver started at http://127.16.86.131:44835/ using document root <none> and password file <none>
I20260806 09:27:08.996178 12934 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:08.996332 12934 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:08.996735 12934 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:09.000602 12934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/data/instance:
uuid: "84dcd433a86e45459048bf45eaf7921e"
format_stamp: "Formatted at 2026-08-06 09:27:08 on dist-test-slave-h3ft"
server_key: "032c705c1aee30f035c00ff7fb65a239"
server_key_iv: "b6aaabb5adc7ffcae65f1a5cd71684cd"
server_key_version: "encryptionkey@0"
I20260806 09:27:09.001675 12934 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance:
uuid: "84dcd433a86e45459048bf45eaf7921e"
format_stamp: "Formatted at 2026-08-06 09:27:08 on dist-test-slave-h3ft"
server_key: "032c705c1aee30f035c00ff7fb65a239"
server_key_iv: "b6aaabb5adc7ffcae65f1a5cd71684cd"
server_key_version: "encryptionkey@0"
I20260806 09:27:09.008917 12934 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260806 09:27:09.014142 12956 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:09.016419 12934 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:27:09.016652 12934 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/wal
uuid: "84dcd433a86e45459048bf45eaf7921e"
format_stamp: "Formatted at 2026-08-06 09:27:08 on dist-test-slave-h3ft"
server_key: "032c705c1aee30f035c00ff7fb65a239"
server_key_iv: "b6aaabb5adc7ffcae65f1a5cd71684cd"
server_key_version: "encryptionkey@0"
I20260806 09:27:09.016932 12934 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:09.038340 12934 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:09.039657 12934 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:09.040021 12934 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:09.041347 12934 txn_system_client.cc:433] TxnSystemClient initialization is disabled...
I20260806 09:27:09.043342 12934 ts_tablet_manager.cc:579] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260806 09:27:09.043454 12934 ts_tablet_manager.cc:525] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:09.043566 12934 ts_tablet_manager.cc:610] Registered 0 tablets
I20260806 09:27:09.043633 12934 ts_tablet_manager.cc:589] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:09.088279 12934 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.131:43481
I20260806 09:27:09.088327 13109 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.131:43481 every 8 connection(s)
I20260806 09:27:09.089692 12934 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/data/info.pb
I20260806 09:27:09.090888 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 12934
I20260806 09:27:09.091182 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/wal/instance
I20260806 09:27:09.091918 16730 external_mini_cluster.cc:1428] Setting key 29065a7630c41ada1fea25ddd14f8813
I20260806 09:27:09.106580 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.093481 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.131:52349 (local address 127.16.86.190:38383)
0806 09:27:09.094017 (+ 536us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.094022 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.096164 (+ 2142us) server_negotiation.cc:407] Connection header received
0806 09:27:09.098477 (+ 2313us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.098482 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.098564 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.098689 (+ 125us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:09.100256 (+ 1567us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.101075 (+ 819us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.103138 (+ 2063us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.103578 (+ 440us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.103698 (+ 120us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.105547 (+ 1849us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:09.105552 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:09.105555 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:09.105577 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:09.105625 (+ 48us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:09.105639 (+ 14us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:09.105727 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.106057 (+ 330us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.106168 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":385,"thread_start_us":279,"threads_started":1}
I20260806 09:27:09.108378 13117 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.094406 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:38383 (local address 127.16.86.131:52349)
0806 09:27:09.095944 (+ 1538us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:09.096013 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:09.098086 (+ 2073us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:09.099062 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:09.099083 (+ 21us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:09.099208 (+ 125us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:09.100052 (+ 844us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.100075 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.101249 (+ 1174us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.101253 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.102882 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.102897 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.103865 (+ 968us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.103872 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.104168 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.105134 (+ 966us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
0806 09:27:09.105137 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:09.105191 (+ 54us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
0806 09:27:09.105297 (+ 106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:09.105778 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:09.105790 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:09.105799 (+ 9us) client_negotiation.cc:769] Sending connection context
0806 09:27:09.106043 (+ 244us) client_negotiation.cc:240] Negotiation successful
0806 09:27:09.106498 (+ 455us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":832,"thread_start_us":547,"threads_started":1}
I20260806 09:27:09.110122 13110 heartbeater.cc:347] Connected to a master server at 127.16.86.190:38383
I20260806 09:27:09.110556 13110 heartbeater.cc:464] Registering TS with master...
I20260806 09:27:09.111728 13110 heartbeater.cc:511] Master 127.16.86.190:38383 requested a full tablet report, sending...
I20260806 09:27:09.113637 12495 ts_manager.cc:194] Registered new tserver with Master: 84dcd433a86e45459048bf45eaf7921e (127.16.86.131:43481)
I20260806 09:27:09.114518 12495 master_service.cc:496] Signed X509 certificate for tserver {username='slave'} at 127.16.86.131:52349
I20260806 09:27:09.122489 16730 external_mini_cluster.cc:930] 3 TS(s) registered with all masters
I20260806 09:27:09.134603 13115 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.127696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51588 (local address 127.16.86.190:38383)
0806 09:27:09.127918 (+ 222us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.127927 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.128723 (+ 796us) server_negotiation.cc:407] Connection header received
0806 09:27:09.128904 (+ 181us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.128908 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.128977 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.129073 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:09.129609 (+ 536us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.130339 (+ 730us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.131965 (+ 1626us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.132381 (+ 416us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.132483 (+ 102us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.133781 (+ 1298us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:09.133786 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:09.133789 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:09.133807 (+ 18us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:09.133845 (+ 38us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:09.133857 (+ 12us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:09.133995 (+ 138us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.134213 (+ 218us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.134332 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:27:09.139863 12495 catalog_manager.cc:1908] Servicing CreateTable request from {username='slave'} at 127.0.0.1:51588:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20260806 09:27:09.147142 12495 catalog_manager.cc:6854] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260806 09:27:09.193216 13136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.163101 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:43481 (local address 127.0.0.1:36224)
0806 09:27:09.178316 (+ 15215us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:09.178353 (+ 37us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:09.178592 (+ 239us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:09.180635 (+ 2043us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:09.180647 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:09.180690 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:27:09.181208 (+ 518us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.181231 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.184204 (+ 2973us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.184213 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.188284 (+ 4071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.188297 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.191999 (+ 3702us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.192007 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.192269 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.192337 (+ 68us) client_negotiation.cc:769] Sending connection context
0806 09:27:09.192494 (+ 157us) client_negotiation.cc:240] Negotiation successful
0806 09:27:09.192796 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14959,"thread_start_us":16725,"threads_started":1}
I20260806 09:27:09.194555 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.163667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36224 (local address 127.16.86.131:43481)
0806 09:27:09.179137 (+ 15470us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.179155 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.179206 (+ 51us) server_negotiation.cc:407] Connection header received
0806 09:27:09.179369 (+ 163us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.179378 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.179731 (+ 353us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.179974 (+ 243us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:27:09.181564 (+ 1590us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.183954 (+ 2390us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.188613 (+ 4659us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.190267 (+ 1654us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.190420 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.193827 (+ 3407us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.193920 (+ 93us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.194127 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15200,"thread_start_us":16560,"threads_started":1}
I20260806 09:27:09.195235 13134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.162697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:42407 (local address 127.0.0.1:41380)
0806 09:27:09.177497 (+ 14800us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:09.177566 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:09.177806 (+ 240us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:09.186722 (+ 8916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:09.186731 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:09.186763 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:27:09.188067 (+ 1304us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.188082 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.190098 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.190105 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.192483 (+ 2378us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.192497 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.194430 (+ 1933us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.194437 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.194659 (+ 222us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.194700 (+ 41us) client_negotiation.cc:769] Sending connection context
0806 09:27:09.194766 (+ 66us) client_negotiation.cc:240] Negotiation successful
0806 09:27:09.194896 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14435,"thread_start_us":15319,"threads_started":1}
I20260806 09:27:09.196141 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.162865 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41380 (local address 127.16.86.130:42407)
0806 09:27:09.185655 (+ 22790us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.185666 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.185703 (+ 37us) server_negotiation.cc:407] Connection header received
0806 09:27:09.185824 (+ 121us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.185832 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.186123 (+ 291us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.186968 (+ 845us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:27:09.188314 (+ 1346us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.189903 (+ 1589us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.192777 (+ 2874us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.194176 (+ 1399us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.195235 (+ 1059us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.195499 (+ 264us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.195608 (+ 109us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.195782 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":22558,"thread_start_us":23431,"threads_started":1}
I20260806 09:27:09.197408 13140 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.186649 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:33797 (local address 127.0.0.1:40274)
0806 09:27:09.188197 (+ 1548us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:09.188220 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:09.188376 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:09.189447 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:09.189452 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:09.189478 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:27:09.189969 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.189981 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.191968 (+ 1987us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.191977 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.194432 (+ 2455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.194447 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.196540 (+ 2093us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.196547 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.196750 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.196782 (+ 32us) client_negotiation.cc:769] Sending connection context
0806 09:27:09.196879 (+ 97us) client_negotiation.cc:240] Negotiation successful
0806 09:27:09.197027 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1394,"mutex_wait_us":2,"thread_start_us":8497,"threads_started":1}
I20260806 09:27:09.198261 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.180821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40274 (local address 127.16.86.129:33797)
0806 09:27:09.187642 (+ 6821us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.187652 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.188513 (+ 861us) server_negotiation.cc:407] Connection header received
0806 09:27:09.188659 (+ 146us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.188668 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.188949 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.189172 (+ 223us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:27:09.190184 (+ 1012us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.191792 (+ 1608us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.194728 (+ 2936us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.196295 (+ 1567us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.196435 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.196733 (+ 298us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.197615 (+ 882us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.197871 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6591,"thread_start_us":6387,"threads_started":1}
I20260806 09:27:09.200661 13020 tablet_service.cc:1467] Processing CreateTablet for tablet eb06c2e3cc564ea79c828434b3af99fe (DEFAULT_TABLE table=test-table [id=ef7bc85edbaa4a468acb88fcaebe9734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:27:09.202658 13020 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb06c2e3cc564ea79c828434b3af99fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:09.202001 12838 tablet_service.cc:1467] Processing CreateTablet for tablet eb06c2e3cc564ea79c828434b3af99fe (DEFAULT_TABLE table=test-table [id=ef7bc85edbaa4a468acb88fcaebe9734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:27:09.203648 12838 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb06c2e3cc564ea79c828434b3af99fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:09.203253 12664 tablet_service.cc:1467] Processing CreateTablet for tablet eb06c2e3cc564ea79c828434b3af99fe (DEFAULT_TABLE table=test-table [id=ef7bc85edbaa4a468acb88fcaebe9734]), partition=RANGE (key) PARTITION UNBOUNDED
I20260806 09:27:09.204885 12664 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet eb06c2e3cc564ea79c828434b3af99fe. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:09.215799 13141 tablet_bootstrap.cc:492] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a: Bootstrap starting.
I20260806 09:27:09.219950 13141 tablet_bootstrap.cc:654] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:09.221066 13143 tablet_bootstrap.cc:492] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e: Bootstrap starting.
I20260806 09:27:09.223058 13141 log.cc:824] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:09.224301 13142 tablet_bootstrap.cc:492] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc: Bootstrap starting.
I20260806 09:27:09.226081 13143 tablet_bootstrap.cc:654] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:09.227483 13143 log.cc:824] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:09.228384 13141 tablet_bootstrap.cc:492] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a: No bootstrap required, opened a new log
I20260806 09:27:09.228654 13142 tablet_bootstrap.cc:654] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:09.228729 13141 ts_tablet_manager.cc:1397] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a: Time spent bootstrapping tablet: real 0.013s user 0.004s sys 0.006s
I20260806 09:27:09.229959 13142 log.cc:824] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:09.231734 13143 tablet_bootstrap.cc:492] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e: No bootstrap required, opened a new log
I20260806 09:27:09.232015 13143 ts_tablet_manager.cc:1397] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260806 09:27:09.233440 13142 tablet_bootstrap.cc:492] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc: No bootstrap required, opened a new log
I20260806 09:27:09.233737 13142 ts_tablet_manager.cc:1397] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20260806 09:27:09.239805 13141 raft_consensus.cc:348] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfff36343a7447abea024b6aace647a" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 42407 } } peers { permanent_uuid: "84dcd433a86e45459048bf45eaf7921e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43481 } } peers { permanent_uuid: "97dcecd806344effba2d54e49295f6bc" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33797 } }
I20260806 09:27:09.240329 13141 raft_consensus.cc:374] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:09.240440 13141 raft_consensus.cc:729] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ecfff36343a7447abea024b6aace647a, State: Initialized, Role: FOLLOWER
I20260806 09:27:09.241137 13141 consensus_queue.cc:260] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfff36343a7447abea024b6aace647a" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 42407 } } peers { permanent_uuid: "84dcd433a86e45459048bf45eaf7921e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43481 } } peers { permanent_uuid: "97dcecd806344effba2d54e49295f6bc" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33797 } }
I20260806 09:27:09.243184 13142 raft_consensus.cc:348] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfff36343a7447abea024b6aace647a" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 42407 } } peers { permanent_uuid: "84dcd433a86e45459048bf45eaf7921e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43481 } } peers { permanent_uuid: "97dcecd806344effba2d54e49295f6bc" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33797 } }
I20260806 09:27:09.243754 13142 raft_consensus.cc:374] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:09.243886 13142 raft_consensus.cc:729] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97dcecd806344effba2d54e49295f6bc, State: Initialized, Role: FOLLOWER
I20260806 09:27:09.243777 13143 raft_consensus.cc:348] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfff36343a7447abea024b6aace647a" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 42407 } } peers { permanent_uuid: "84dcd433a86e45459048bf45eaf7921e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43481 } } peers { permanent_uuid: "97dcecd806344effba2d54e49295f6bc" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33797 } }
I20260806 09:27:09.244136 12930 heartbeater.cc:503] Master 127.16.86.190:38383 was elected leader, sending a full tablet report...
I20260806 09:27:09.244256 13143 raft_consensus.cc:374] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:09.244364 13143 raft_consensus.cc:729] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84dcd433a86e45459048bf45eaf7921e, State: Initialized, Role: FOLLOWER
I20260806 09:27:09.244719 13141 ts_tablet_manager.cc:1428] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a: Time spent starting tablet: real 0.016s user 0.000s sys 0.016s
I20260806 09:27:09.244959 13142 consensus_queue.cc:260] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfff36343a7447abea024b6aace647a" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 42407 } } peers { permanent_uuid: "84dcd433a86e45459048bf45eaf7921e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43481 } } peers { permanent_uuid: "97dcecd806344effba2d54e49295f6bc" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33797 } }
I20260806 09:27:09.245008 13143 consensus_queue.cc:260] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfff36343a7447abea024b6aace647a" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 42407 } } peers { permanent_uuid: "84dcd433a86e45459048bf45eaf7921e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43481 } } peers { permanent_uuid: "97dcecd806344effba2d54e49295f6bc" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33797 } }
W20260806 09:27:09.245291 12931 tablet.cc:2369] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:27:09.248332 13142 ts_tablet_manager.cc:1428] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc: Time spent starting tablet: real 0.014s user 0.015s sys 0.000s
I20260806 09:27:09.249979 13110 heartbeater.cc:503] Master 127.16.86.190:38383 was elected leader, sending a full tablet report...
I20260806 09:27:09.250701 13143 ts_tablet_manager.cc:1428] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e: Time spent starting tablet: real 0.019s user 0.014s sys 0.004s
W20260806 09:27:09.341723 13111 tablet.cc:2369] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:27:09.397221 13147 raft_consensus.cc:482] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:27:09.397646 13147 raft_consensus.cc:504] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfff36343a7447abea024b6aace647a" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 42407 } } peers { permanent_uuid: "84dcd433a86e45459048bf45eaf7921e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43481 } } peers { permanent_uuid: "97dcecd806344effba2d54e49295f6bc" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33797 } }
I20260806 09:27:09.399580 13147 leader_election.cc:288] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 84dcd433a86e45459048bf45eaf7921e (127.16.86.131:43481), 97dcecd806344effba2d54e49295f6bc (127.16.86.129:33797)
I20260806 09:27:09.410128 13150 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.400315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:43481 (local address 127.16.86.130:40491)
0806 09:27:09.401062 (+ 747us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:09.401081 (+ 19us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:09.401318 (+ 237us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:09.401900 (+ 582us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:09.401910 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:09.401940 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:27:09.402361 (+ 421us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.402370 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.403945 (+ 1575us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.403950 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.406890 (+ 2940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.406900 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.408923 (+ 2023us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.408932 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.409172 (+ 240us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.409279 (+ 107us) client_negotiation.cc:769] Sending connection context
0806 09:27:09.409392 (+ 113us) client_negotiation.cc:240] Negotiation successful
0806 09:27:09.409658 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":617,"mutex_wait_us":170,"thread_start_us":421,"threads_started":1}
I20260806 09:27:09.410128 13151 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.401241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.129:33797 (local address 127.16.86.130:44465)
0806 09:27:09.401921 (+ 680us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:09.401938 (+ 17us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:09.402091 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:09.402596 (+ 505us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:09.402601 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:09.402626 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:27:09.402907 (+ 281us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.402915 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.404275 (+ 1360us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.404283 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.406881 (+ 2598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.406899 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.408923 (+ 2024us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.408932 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.409176 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.409277 (+ 101us) client_negotiation.cc:769] Sending connection context
0806 09:27:09.409392 (+ 115us) client_negotiation.cc:240] Negotiation successful
0806 09:27:09.409658 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"mutex_wait_us":209,"thread_start_us":349,"threads_started":1}
I20260806 09:27:09.411126 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.400303 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:40491 (local address 127.16.86.131:43481)
0806 09:27:09.400560 (+ 257us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.400566 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.401222 (+ 656us) server_negotiation.cc:407] Connection header received
0806 09:27:09.401573 (+ 351us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.401582 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.401725 (+ 143us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.401855 (+ 130us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:27:09.402544 (+ 689us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.403769 (+ 1225us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.407229 (+ 3460us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.408584 (+ 1355us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.408782 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.408867 (+ 85us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.410137 (+ 1270us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.410802 (+ 665us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:27:09.411175 13148 raft_consensus.cc:482] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260806 09:27:09.411808 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.401480 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.130:44465 (local address 127.16.86.129:33797)
0806 09:27:09.401741 (+ 261us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.401749 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.402023 (+ 274us) server_negotiation.cc:407] Connection header received
0806 09:27:09.402323 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.402328 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.402417 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.402577 (+ 160us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:27:09.403125 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.404113 (+ 988us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.407186 (+ 3073us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.408584 (+ 1398us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.408782 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.408869 (+ 87us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.409515 (+ 646us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.409869 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:27:09.411522 13148 raft_consensus.cc:504] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfff36343a7447abea024b6aace647a" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 42407 } } peers { permanent_uuid: "84dcd433a86e45459048bf45eaf7921e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43481 } } peers { permanent_uuid: "97dcecd806344effba2d54e49295f6bc" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33797 } }
I20260806 09:27:09.412393 13045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb06c2e3cc564ea79c828434b3af99fe" candidate_uuid: "ecfff36343a7447abea024b6aace647a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84dcd433a86e45459048bf45eaf7921e" is_pre_election: true
I20260806 09:27:09.412904 12690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb06c2e3cc564ea79c828434b3af99fe" candidate_uuid: "ecfff36343a7447abea024b6aace647a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dcecd806344effba2d54e49295f6bc" is_pre_election: true
I20260806 09:27:09.413121 13045 raft_consensus.cc:2454] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecfff36343a7447abea024b6aace647a in term 0.
I20260806 09:27:09.413368 12690 raft_consensus.cc:2454] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ecfff36343a7447abea024b6aace647a in term 0.
I20260806 09:27:09.413928 13148 leader_election.cc:288] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ecfff36343a7447abea024b6aace647a (127.16.86.130:42407), 84dcd433a86e45459048bf45eaf7921e (127.16.86.131:43481)
I20260806 09:27:09.414008 12777 leader_election.cc:302] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84dcd433a86e45459048bf45eaf7921e, ecfff36343a7447abea024b6aace647a; no voters:
I20260806 09:27:09.414970 13147 raft_consensus.cc:2781] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260806 09:27:09.415184 13147 raft_consensus.cc:482] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260806 09:27:09.415297 13147 raft_consensus.cc:3037] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:09.420029 13147 raft_consensus.cc:504] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfff36343a7447abea024b6aace647a" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 42407 } } peers { permanent_uuid: "84dcd433a86e45459048bf45eaf7921e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43481 } } peers { permanent_uuid: "97dcecd806344effba2d54e49295f6bc" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33797 } }
I20260806 09:27:09.420964 13147 leader_election.cc:288] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [CANDIDATE]: Term 1 election: Requested vote from peers 84dcd433a86e45459048bf45eaf7921e (127.16.86.131:43481), 97dcecd806344effba2d54e49295f6bc (127.16.86.129:33797)
I20260806 09:27:09.421625 12690 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb06c2e3cc564ea79c828434b3af99fe" candidate_uuid: "ecfff36343a7447abea024b6aace647a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97dcecd806344effba2d54e49295f6bc"
I20260806 09:27:09.421638 13045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb06c2e3cc564ea79c828434b3af99fe" candidate_uuid: "ecfff36343a7447abea024b6aace647a" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84dcd433a86e45459048bf45eaf7921e"
I20260806 09:27:09.421900 12690 raft_consensus.cc:3037] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:09.421912 13045 raft_consensus.cc:3037] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:09.422600 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.414740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:33093 (local address 127.16.86.130:42407)
0806 09:27:09.414966 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.414975 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.415773 (+ 798us) server_negotiation.cc:407] Connection header received
0806 09:27:09.416014 (+ 241us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.416020 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.416089 (+ 69us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.416303 (+ 214us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:27:09.416851 (+ 548us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.417645 (+ 794us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.419973 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.421176 (+ 1203us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.421355 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.421443 (+ 88us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.422066 (+ 623us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.422218 (+ 152us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20260806 09:27:09.423411 13154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.414456 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.130:42407 (local address 127.16.86.129:33093)
0806 09:27:09.415663 (+ 1207us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:09.415679 (+ 16us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:09.415849 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:09.416319 (+ 470us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:09.416323 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:09.416348 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:27:09.416707 (+ 359us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.416716 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.417793 (+ 1077us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.417798 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.419703 (+ 1905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.419715 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.421624 (+ 1909us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.421631 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.421855 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.421900 (+ 45us) client_negotiation.cc:769] Sending connection context
0806 09:27:09.422867 (+ 967us) client_negotiation.cc:240] Negotiation successful
0806 09:27:09.423047 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1063,"thread_start_us":4810,"threads_started":1}
I20260806 09:27:09.425765 13045 raft_consensus.cc:2454] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecfff36343a7447abea024b6aace647a in term 1.
I20260806 09:27:09.426105 12690 raft_consensus.cc:2454] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ecfff36343a7447abea024b6aace647a in term 1.
I20260806 09:27:09.426467 12777 leader_election.cc:302] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84dcd433a86e45459048bf45eaf7921e, ecfff36343a7447abea024b6aace647a; no voters:
I20260806 09:27:09.426928 13147 raft_consensus.cc:2781] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:09.427596 12867 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb06c2e3cc564ea79c828434b3af99fe" candidate_uuid: "97dcecd806344effba2d54e49295f6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ecfff36343a7447abea024b6aace647a" is_pre_election: true
W20260806 09:27:09.427958 12754 tablet.cc:2369] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:27:09.428540 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.415062 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.16.86.129:44995 (local address 127.16.86.131:43481)
0806 09:27:09.420298 (+ 5236us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.420305 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.420328 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:27:09.420412 (+ 84us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.420416 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.420512 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.420649 (+ 137us) server_negotiation.cc:226] Negotiated authn=CERTIFICATE
0806 09:27:09.421465 (+ 816us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.423427 (+ 1962us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.425755 (+ 2328us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.427075 (+ 1320us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.427266 (+ 191us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.427324 (+ 58us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.428042 (+ 718us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.428248 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5072}
I20260806 09:27:09.429010 13147 raft_consensus.cc:686] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 1 LEADER]: Becoming Leader. State: Replica: ecfff36343a7447abea024b6aace647a, State: Running, Role: LEADER
I20260806 09:27:09.429247 13152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.414486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.131:43481 (local address 127.16.86.129:44995)
0806 09:27:09.419555 (+ 5069us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:09.419576 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:09.419703 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:09.420686 (+ 983us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:09.420691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:09.420718 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
0806 09:27:09.421210 (+ 492us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.421227 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.423797 (+ 2570us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.423800 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.425551 (+ 1751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:09.425565 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.427477 (+ 1912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.427481 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:09.427596 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.427659 (+ 63us) client_negotiation.cc:769] Sending connection context
0806 09:27:09.427757 (+ 98us) client_negotiation.cc:240] Negotiation successful
0806 09:27:09.427900 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4934,"thread_start_us":784,"threads_started":1}
I20260806 09:27:09.429659 13147 consensus_queue.cc:237] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfff36343a7447abea024b6aace647a" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 42407 } } peers { permanent_uuid: "84dcd433a86e45459048bf45eaf7921e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43481 } } peers { permanent_uuid: "97dcecd806344effba2d54e49295f6bc" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33797 } }
I20260806 09:27:09.429960 13045 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "eb06c2e3cc564ea79c828434b3af99fe" candidate_uuid: "97dcecd806344effba2d54e49295f6bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84dcd433a86e45459048bf45eaf7921e" is_pre_election: true
I20260806 09:27:09.430312 13045 raft_consensus.cc:2379] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97dcecd806344effba2d54e49295f6bc in current term 1: Already voted for candidate ecfff36343a7447abea024b6aace647a in this term.
I20260806 09:27:09.434718 12594 leader_election.cc:302] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97dcecd806344effba2d54e49295f6bc; no voters: 84dcd433a86e45459048bf45eaf7921e, ecfff36343a7447abea024b6aace647a
I20260806 09:27:09.435361 13148 raft_consensus.cc:2726] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260806 09:27:09.436019 12493 catalog_manager.cc:5495] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a reported cstate change: term changed from 0 to 1, leader changed from <none> to ecfff36343a7447abea024b6aace647a (127.16.86.130). New cstate: current_term: 1 leader_uuid: "ecfff36343a7447abea024b6aace647a" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ecfff36343a7447abea024b6aace647a" member_type: VOTER last_known_addr { host: "127.16.86.130" port: 42407 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "84dcd433a86e45459048bf45eaf7921e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43481 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "97dcecd806344effba2d54e49295f6bc" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33797 } health_report { overall_health: UNKNOWN } } }
I20260806 09:27:09.485607 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.479411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41392 (local address 127.16.86.130:42407)
0806 09:27:09.479636 (+ 225us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.479641 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.479663 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:27:09.479870 (+ 207us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.479874 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.479991 (+ 117us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.480152 (+ 161us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:27:09.480603 (+ 451us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.481431 (+ 828us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.482997 (+ 1566us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.483511 (+ 514us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.483629 (+ 118us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.484258 (+ 629us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:27:09.484836 (+ 578us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:27:09.484966 (+ 130us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.485169 (+ 203us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.485241 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 09:27:09.497131 13045 raft_consensus.cc:1261] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e [term 1 FOLLOWER]: Refusing update from remote peer ecfff36343a7447abea024b6aace647a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:27:09.498283 13147 consensus_queue.cc:1035] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [LEADER]: Connected to new peer: Peer: permanent_uuid: "84dcd433a86e45459048bf45eaf7921e" member_type: VOTER last_known_addr { host: "127.16.86.131" port: 43481 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:27:09.498572 12690 raft_consensus.cc:1261] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [term 1 FOLLOWER]: Refusing update from remote peer ecfff36343a7447abea024b6aace647a: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:27:09.499538 13147 consensus_queue.cc:1035] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [LEADER]: Connected to new peer: Peer: permanent_uuid: "97dcecd806344effba2d54e49295f6bc" member_type: VOTER last_known_addr { host: "127.16.86.129" port: 33797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:27:09.518721 13161 mvcc.cc:205] Tried to move back new op lower bound from 7315490527206969344 to 7315490526949609472. Current Snapshot: MvccSnapshot[applied={T|T < 7315490527206969344}]
I20260806 09:27:09.531916 12494 catalog_manager.cc:2461] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:51588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:27:09.532407 12494 catalog_manager.cc:2645] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:51588:
table { table_name: "test-table" } modify_external_catalogs: true
I20260806 09:27:09.537751 12494 catalog_manager.cc:5782] T 00000000000000000000000000000000 P a2f734d5f35a44cd9ffffa58a0ea4dfb: Sending DeleteTablet for 3 replicas of tablet eb06c2e3cc564ea79c828434b3af99fe
I20260806 09:27:09.539256 13020 tablet_service.cc:1514] Processing DeleteTablet for tablet eb06c2e3cc564ea79c828434b3af99fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:27:09 UTC) from {username='slave'} at 127.0.0.1:36224
I20260806 09:27:09.540083 12838 tablet_service.cc:1514] Processing DeleteTablet for tablet eb06c2e3cc564ea79c828434b3af99fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:27:09 UTC) from {username='slave'} at 127.0.0.1:41380
I20260806 09:27:09.540277 13171 tablet_replica.cc:330] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e: stopping tablet replica
I20260806 09:27:09.540781 13171 raft_consensus.cc:2229] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:27:09.541463 13171 raft_consensus.cc:2258] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:27:09.542037 12664 tablet_service.cc:1514] Processing DeleteTablet for tablet eb06c2e3cc564ea79c828434b3af99fe with delete_type TABLET_DATA_DELETED (Table deleted at 2026-08-06 09:27:09 UTC) from {username='slave'} at 127.0.0.1:40274
I20260806 09:27:09.543478 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 12571
I20260806 09:27:09.544098 13171 ts_tablet_manager.cc:1905] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:27:09.549738 13172 tablet_replica.cc:330] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a: stopping tablet replica
I20260806 09:27:09.550195 13172 raft_consensus.cc:2229] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 1 LEADER]: Raft consensus shutting down.
I20260806 09:27:09.550910 13172 raft_consensus.cc:2258] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:27:09.552445 13171 ts_tablet_manager.cc:1918] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:27:09.552649 13171 log.cc:1196] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-2/wal/wals/eb06c2e3cc564ea79c828434b3af99fe
I20260806 09:27:09.553215 13171 ts_tablet_manager.cc:1939] T eb06c2e3cc564ea79c828434b3af99fe P 84dcd433a86e45459048bf45eaf7921e: Deleting consensus metadata
I20260806 09:27:09.553351 13172 ts_tablet_manager.cc:1905] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:27:09.555384 12476 catalog_manager.cc:4841] TS 84dcd433a86e45459048bf45eaf7921e (127.16.86.131:43481): tablet eb06c2e3cc564ea79c828434b3af99fe (table test-table [id=ef7bc85edbaa4a468acb88fcaebe9734]) successfully deleted
I20260806 09:27:09.556981 13173 tablet_replica.cc:330] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc: stopping tablet replica
I20260806 09:27:09.557598 13173 raft_consensus.cc:2229] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260806 09:27:09.558383 13173 raft_consensus.cc:2258] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260806 09:27:09.560192 13172 ts_tablet_manager.cc:1918] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:27:09.560384 13172 log.cc:1196] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-1/wal/wals/eb06c2e3cc564ea79c828434b3af99fe
I20260806 09:27:09.560849 13172 ts_tablet_manager.cc:1939] T eb06c2e3cc564ea79c828434b3af99fe P ecfff36343a7447abea024b6aace647a: Deleting consensus metadata
I20260806 09:27:09.561187 13173 ts_tablet_manager.cc:1905] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260806 09:27:09.562451 12477 catalog_manager.cc:4841] TS ecfff36343a7447abea024b6aace647a (127.16.86.130:42407): tablet eb06c2e3cc564ea79c828434b3af99fe (table test-table [id=ef7bc85edbaa4a468acb88fcaebe9734]) successfully deleted
I20260806 09:27:09.567610 13173 ts_tablet_manager.cc:1918] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20260806 09:27:09.567858 13173 log.cc:1196] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc: Deleting WAL directory at /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1786008061536593-16730-0/minicluster-data/ts-0/wal/wals/eb06c2e3cc564ea79c828434b3af99fe
I20260806 09:27:09.568440 13173 ts_tablet_manager.cc:1939] T eb06c2e3cc564ea79c828434b3af99fe P 97dcecd806344effba2d54e49295f6bc: Deleting consensus metadata
I20260806 09:27:09.569453 13139 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.546524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40280 (local address 127.16.86.129:33797)
0806 09:27:09.546744 (+ 220us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.546750 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.554877 (+ 8127us) server_negotiation.cc:407] Connection header received
0806 09:27:09.556534 (+ 1657us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.556543 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.556744 (+ 201us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.556846 (+ 102us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:09.557684 (+ 838us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.558965 (+ 1281us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.561014 (+ 2049us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.561678 (+ 664us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.561785 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.567356 (+ 5571us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:09.567371 (+ 15us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:09.567392 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:09.568343 (+ 951us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:09.568421 (+ 78us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:09.568457 (+ 36us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:09.568619 (+ 162us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.568865 (+ 246us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.569026 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":95}
I20260806 09:27:09.570358 12746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:09.570845 12476 catalog_manager.cc:4841] TS 97dcecd806344effba2d54e49295f6bc (127.16.86.129:33797): tablet eb06c2e3cc564ea79c828434b3af99fe (table test-table [id=ef7bc85edbaa4a468acb88fcaebe9734]) successfully deleted
I20260806 09:27:09.677294 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 12571
I20260806 09:27:09.712199 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 12757
I20260806 09:27:09.721401 13135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.713260 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41408 (local address 127.16.86.130:42407)
0806 09:27:09.713486 (+ 226us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.713493 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.713525 (+ 32us) server_negotiation.cc:407] Connection header received
0806 09:27:09.713701 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.713707 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.713847 (+ 140us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.713982 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:09.714435 (+ 453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.715606 (+ 1171us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.717112 (+ 1506us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.717806 (+ 694us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.717954 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.719519 (+ 1565us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:09.719532 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:09.719553 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:09.720448 (+ 895us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:09.720528 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:09.720555 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:09.720676 (+ 121us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.720877 (+ 201us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.720988 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":93}
I20260806 09:27:09.722520 12921 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:09.848491 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 12757
I20260806 09:27:09.879822 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 12934
I20260806 09:27:09.887856 13137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:09.880837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36232 (local address 127.16.86.131:43481)
0806 09:27:09.881056 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:09.881061 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:09.881095 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:27:09.881307 (+ 212us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:09.881313 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:09.881420 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:09.881553 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:09.882060 (+ 507us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.882883 (+ 823us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.884424 (+ 1541us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:09.884879 (+ 455us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:09.885002 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:09.886259 (+ 1257us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:09.886267 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:09.886282 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:09.886919 (+ 637us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:09.886973 (+ 54us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:09.886994 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:09.887227 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:09.887439 (+ 212us) server_negotiation.cc:299] Negotiation successful
0806 09:27:09.887527 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:27:09.888656 13099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:09.991348 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 12934
I20260806 09:27:10.020047 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 12453
I20260806 09:27:10.021555 12537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:10.136178 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 12453
2026-08-06T09:27:10Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (3669 ms)
[----------] 6 tests from EncryptionPolicyTest (27071 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:10 dist-test-slave-h3ft krb5kdc[13184](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:10 dist-test-slave-h3ft krb5kdc[13184](info): set up 2 sockets
Aug 06 09:27:10 dist-test-slave-h3ft krb5kdc[13184](info): commencing operation
krb5kdc: starting...
W20260806 09:27:11.540459 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.341s user 0.003s sys 0.052s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:11 dist-test-slave-h3ft krb5kdc[13184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:11Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:27:11.948278 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40683
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44219
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40683
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:12.359597 13202 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:12.359925 13202 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:12.360020 13202 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:12.367403 13202 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:12.367539 13202 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:12.367597 13202 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:12.367667 13202 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:12.367720 13202 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:12.378170 13202 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44219
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40683
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40683
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:12.379688 13202 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:12.381068 13202 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:12.395117 13211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:12.395130 13215 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:12.395411 13212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:12.397423 13202 server_base.cc:1034] running on GCE node
I20260806 09:27:12.398679 13202 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:12.400714 13202 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:12.402011 13202 hybrid_clock.cc:648] HybridClock initialized: now 1786008432401940 us; error 56 us; skew 500 ppm
I20260806 09:27:12.402495 13202 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:12 dist-test-slave-h3ft krb5kdc[13184](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008432, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:12.406950 13202 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:27:12.408963 13202 webserver.cc:460] Webserver started at http://127.0.0.1:46125/ using document root <none> and password file <none>
I20260806 09:27:12.409857 13202 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:12.410033 13202 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:12.410398 13202 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:12.414318 13202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "d877ae52395244bcbe7d6afe6ebcc6f2"
format_stamp: "Formatted at 2026-08-06 09:27:12 on dist-test-slave-h3ft"
server_key: "9fec0c809f9bfba6b84c319fee8aebb1"
server_key_iv: "7af5dec1705b64513f213d00016bbfeb"
server_key_version: "encryptionkey@0"
I20260806 09:27:12.415424 13202 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "d877ae52395244bcbe7d6afe6ebcc6f2"
format_stamp: "Formatted at 2026-08-06 09:27:12 on dist-test-slave-h3ft"
server_key: "9fec0c809f9bfba6b84c319fee8aebb1"
server_key_iv: "7af5dec1705b64513f213d00016bbfeb"
server_key_version: "encryptionkey@0"
I20260806 09:27:12.422430 13202 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260806 09:27:12.427278 13224 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:12.429157 13202 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:27:12.429404 13202 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "d877ae52395244bcbe7d6afe6ebcc6f2"
format_stamp: "Formatted at 2026-08-06 09:27:12 on dist-test-slave-h3ft"
server_key: "9fec0c809f9bfba6b84c319fee8aebb1"
server_key_iv: "7af5dec1705b64513f213d00016bbfeb"
server_key_version: "encryptionkey@0"
I20260806 09:27:12.429654 13202 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:12.451370 13202 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:12.456776 13202 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:12.457216 13202 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:12.481328 13202 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40683
I20260806 09:27:12.481388 13299 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40683 every 8 connection(s)
I20260806 09:27:12.483255 13202 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:12.487902 13301 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:12.489622 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 13202
I20260806 09:27:12.489938 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:12.490734 16730 external_mini_cluster.cc:1428] Setting key b5c626aab5b1d18c92661bb5c4a0c19b
I20260806 09:27:12.501780 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2: Bootstrap starting.
I20260806 09:27:12.505896 13301 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:12.507427 13301 log.cc:824] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:27:12 dist-test-slave-h3ft krb5kdc[13184](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008431, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 09:27:12.511612 13301 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2: No bootstrap required, opened a new log
I20260806 09:27:12.521879 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:12.493533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35925 (local address 127.0.0.1:40683)
0806 09:27:12.494685 (+ 1152us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:12.494700 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:12.494753 (+ 53us) server_negotiation.cc:407] Connection header received
0806 09:27:12.497208 (+ 2455us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:12.497241 (+ 33us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:12.497802 (+ 561us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:12.498275 (+ 473us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:12.500345 (+ 2070us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:12.502218 (+ 1873us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:12.504275 (+ 2057us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:12.505199 (+ 924us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:12.505403 (+ 204us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:12.511560 (+ 6157us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:12.511574 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:12.511606 (+ 32us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:12.511745 (+ 139us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:12.516370 (+ 4625us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:12.517620 (+ 1250us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:12.517627 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:12.517636 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:12.517853 (+ 217us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:12.518393 (+ 540us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:12.518397 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:12.518406 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:12.519192 (+ 786us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:12.519514 (+ 322us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:12.520078 (+ 564us) server_negotiation.cc:299] Negotiation successful
0806 09:27:12.520542 (+ 464us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":844,"thread_start_us":477,"threads_started":1}
I20260806 09:27:12.521778 13301 raft_consensus.cc:348] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d877ae52395244bcbe7d6afe6ebcc6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40683 } }
I20260806 09:27:12.522316 13301 raft_consensus.cc:374] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:12.522455 13301 raft_consensus.cc:729] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d877ae52395244bcbe7d6afe6ebcc6f2, State: Initialized, Role: FOLLOWER
I20260806 09:27:12.523204 13301 consensus_queue.cc:260] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d877ae52395244bcbe7d6afe6ebcc6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40683 } }
I20260806 09:27:12.523471 13301 raft_consensus.cc:388] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:12.523650 13301 raft_consensus.cc:482] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:12.523842 13301 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:12.526538 13301 raft_consensus.cc:504] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d877ae52395244bcbe7d6afe6ebcc6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40683 } }
I20260806 09:27:12.526968 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:12.527226 13301 leader_election.cc:302] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d877ae52395244bcbe7d6afe6ebcc6f2; no voters:
I20260806 09:27:12.527904 13301 leader_election.cc:288] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:12.528151 13307 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:12.530184 13301 sys_catalog.cc:564] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:12.534929 13307 raft_consensus.cc:686] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [term 1 LEADER]: Becoming Leader. State: Replica: d877ae52395244bcbe7d6afe6ebcc6f2, State: Running, Role: LEADER
I20260806 09:27:12.535761 13307 consensus_queue.cc:237] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d877ae52395244bcbe7d6afe6ebcc6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40683 } }
I20260806 09:27:12.540198 13309 sys_catalog.cc:455] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d877ae52395244bcbe7d6afe6ebcc6f2. Latest consensus state: current_term: 1 leader_uuid: "d877ae52395244bcbe7d6afe6ebcc6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d877ae52395244bcbe7d6afe6ebcc6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40683 } } }
I20260806 09:27:12.539739 13308 sys_catalog.cc:455] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d877ae52395244bcbe7d6afe6ebcc6f2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d877ae52395244bcbe7d6afe6ebcc6f2" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40683 } } }
I20260806 09:27:12.542348 13309 sys_catalog.cc:458] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:12.542348 13308 sys_catalog.cc:458] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:12.543458 13330 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:12.556905 13330 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:12.562681 13304 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:12.535443 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54483 (local address 127.0.0.1:40683)
0806 09:27:12.535698 (+ 255us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:12.535706 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:12.537272 (+ 1566us) server_negotiation.cc:407] Connection header received
0806 09:27:12.537526 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:12.537534 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:12.537656 (+ 122us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:12.537813 (+ 157us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:12.540476 (+ 2663us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:12.541763 (+ 1287us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:12.544585 (+ 2822us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:12.545931 (+ 1346us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:12.546134 (+ 203us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:12.551310 (+ 5176us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:12.551321 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:12.551330 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:12.551395 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:12.556841 (+ 5446us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:12.558296 (+ 1455us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:12.558304 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:12.558311 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:12.558513 (+ 202us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:12.559274 (+ 761us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:12.559282 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:12.559288 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:12.559917 (+ 629us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:12.560300 (+ 383us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:12.561958 (+ 1658us) server_negotiation.cc:299] Negotiation successful
0806 09:27:12.562218 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20260806 09:27:12.569202 13330 catalog_manager.cc:1347] Generated new cluster ID: abf4d21d27634ab78035dcdb8e21f60f
I20260806 09:27:12.569340 13330 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:12.586337 13330 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:12.588609 13330 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:12.601876 13330 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d877ae52395244bcbe7d6afe6ebcc6f2: Generated new TSK 0
I20260806 09:27:12.603292 13330 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:12.618490 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 13202
I20260806 09:27:12.620175 13292 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:12.716578 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 13202
W20260806 09:27:12.750674 13182 connection.cc:535] client connection to 127.0.0.1:40683 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:40683): Connection reset by peer (error 104)
2026-08-06T09:27:12Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2573 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:12 dist-test-slave-h3ft krb5kdc[13345](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:12 dist-test-slave-h3ft krb5kdc[13345](info): set up 2 sockets
Aug 06 09:27:12 dist-test-slave-h3ft krb5kdc[13345](info): commencing operation
krb5kdc: starting...
W20260806 09:27:14.090118 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.318s user 0.004s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:14 dist-test-slave-h3ft krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008434, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:14Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:14Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:27:14.495066 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:33459
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:44593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:14.877671 13363 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:14.877998 13363 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:14.878094 13363 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:14.885464 13363 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:14.885608 13363 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:14.885658 13363 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:14.885713 13363 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:14.885764 13363 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:14.895793 13363 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33459
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:44593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:44593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:14.897192 13363 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:14.898481 13363 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:14.912619 13368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:14.912704 13371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:14.912704 13369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:14.914454 13363 server_base.cc:1034] running on GCE node
I20260806 09:27:14.915547 13363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:14.917199 13363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:14.918435 13363 hybrid_clock.cc:648] HybridClock initialized: now 1786008434918379 us; error 60 us; skew 500 ppm
I20260806 09:27:14.918881 13363 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:14 dist-test-slave-h3ft krb5kdc[13345](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008434, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:14.928292 13363 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:27:14.930284 13363 webserver.cc:460] Webserver started at http://127.0.0.1:45445/ using document root <none> and password file <none>
I20260806 09:27:14.931185 13363 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:14.931336 13363 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:14.931697 13363 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:14.934944 13363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "dc3224232c314c69b9e6e86d155d8ef5"
format_stamp: "Formatted at 2026-08-06 09:27:14 on dist-test-slave-h3ft"
server_key: "32f0f7be4c7a33503fcba59249f81c4b"
server_key_iv: "b285c4d600c082877e29229f06b0f6fa"
server_key_version: "encryptionkey@0"
I20260806 09:27:14.935896 13363 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "dc3224232c314c69b9e6e86d155d8ef5"
format_stamp: "Formatted at 2026-08-06 09:27:14 on dist-test-slave-h3ft"
server_key: "32f0f7be4c7a33503fcba59249f81c4b"
server_key_iv: "b285c4d600c082877e29229f06b0f6fa"
server_key_version: "encryptionkey@0"
I20260806 09:27:14.942122 13363 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260806 09:27:14.946763 13384 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:14.948836 13363 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260806 09:27:14.949052 13363 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "dc3224232c314c69b9e6e86d155d8ef5"
format_stamp: "Formatted at 2026-08-06 09:27:14 on dist-test-slave-h3ft"
server_key: "32f0f7be4c7a33503fcba59249f81c4b"
server_key_iv: "b285c4d600c082877e29229f06b0f6fa"
server_key_version: "encryptionkey@0"
I20260806 09:27:14.949342 13363 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:14.969324 13363 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:14.974009 13363 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:14.974345 13363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:14.997130 13363 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:44593
I20260806 09:27:14.997252 13451 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:44593 every 8 connection(s)
I20260806 09:27:14.998642 13363 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:15.000344 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 13363
I20260806 09:27:15.000576 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:15.001359 16730 external_mini_cluster.cc:1428] Setting key 18dadd946650197a15e18fb863d23661
I20260806 09:27:15.004875 13452 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:15.017103 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:15.003808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54143 (local address 127.0.0.1:44593)
0806 09:27:15.004695 (+ 887us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:15.004709 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:15.004762 (+ 53us) server_negotiation.cc:407] Connection header received
0806 09:27:15.006571 (+ 1809us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:15.006595 (+ 24us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:15.007079 (+ 484us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:15.007451 (+ 372us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:15.008291 (+ 840us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:15.009722 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:15.011829 (+ 2107us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:15.012635 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:15.012808 (+ 173us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:15.014436 (+ 1628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:15.014444 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:15.014464 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:15.015129 (+ 665us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:15.015243 (+ 114us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:15.015282 (+ 39us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:15.015417 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:15.015753 (+ 336us) server_negotiation.cc:299] Negotiation successful
0806 09:27:15.015981 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":647,"thread_start_us":396,"threads_started":1}
I20260806 09:27:15.020582 13452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5: Bootstrap starting.
I20260806 09:27:15.021497 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:15.025395 13452 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:15.027180 13452 log.cc:824] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:15.031908 13452 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5: No bootstrap required, opened a new log
I20260806 09:27:15.036190 13457 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:15.028482 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53825 (local address 127.0.0.1:44593)
0806 09:27:15.028664 (+ 182us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:15.028671 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:15.028915 (+ 244us) server_negotiation.cc:407] Connection header received
0806 09:27:15.029183 (+ 268us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:15.029187 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:15.029284 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:15.029405 (+ 121us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:15.030058 (+ 653us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:15.031303 (+ 1245us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:15.033049 (+ 1746us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:15.033760 (+ 711us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:15.033920 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:15.035281 (+ 1361us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:15.035288 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:15.035293 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:15.035319 (+ 26us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:15.035388 (+ 69us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:15.035410 (+ 22us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:15.035549 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:15.035730 (+ 181us) server_negotiation.cc:299] Negotiation successful
0806 09:27:15.035809 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20260806 09:27:15.042147 13452 raft_consensus.cc:348] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3224232c314c69b9e6e86d155d8ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } }
I20260806 09:27:15.042686 13452 raft_consensus.cc:374] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:15.042977 13452 raft_consensus.cc:729] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dc3224232c314c69b9e6e86d155d8ef5, State: Initialized, Role: FOLLOWER
I20260806 09:27:15.043946 13452 consensus_queue.cc:260] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3224232c314c69b9e6e86d155d8ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } }
I20260806 09:27:15.044339 13452 raft_consensus.cc:388] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:15.044536 13452 raft_consensus.cc:482] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:15.044751 13452 raft_consensus.cc:3037] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:15.047673 13452 raft_consensus.cc:504] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3224232c314c69b9e6e86d155d8ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } }
I20260806 09:27:15.048398 13452 leader_election.cc:302] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dc3224232c314c69b9e6e86d155d8ef5; no voters:
I20260806 09:27:15.049476 13452 leader_election.cc:288] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:15.049906 13465 raft_consensus.cc:2781] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:15.051813 13465 raft_consensus.cc:686] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [term 1 LEADER]: Becoming Leader. State: Replica: dc3224232c314c69b9e6e86d155d8ef5, State: Running, Role: LEADER
I20260806 09:27:15.052659 13465 consensus_queue.cc:237] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3224232c314c69b9e6e86d155d8ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } }
I20260806 09:27:15.053131 13452 sys_catalog.cc:564] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:15.056707 13466 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dc3224232c314c69b9e6e86d155d8ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3224232c314c69b9e6e86d155d8ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } } }
I20260806 09:27:15.056766 13468 sys_catalog.cc:455] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dc3224232c314c69b9e6e86d155d8ef5. Latest consensus state: current_term: 1 leader_uuid: "dc3224232c314c69b9e6e86d155d8ef5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dc3224232c314c69b9e6e86d155d8ef5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 44593 } } }
I20260806 09:27:15.057394 13466 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:15.057405 13468 sys_catalog.cc:458] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:15.058493 13473 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:15.065738 13473 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:15.079689 13473 catalog_manager.cc:1347] Generated new cluster ID: d9e2d7fb09974a9eaedd8f54b5979157
I20260806 09:27:15.079877 13473 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:15.110075 13473 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:15.111910 13473 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:15.121047 13473 catalog_manager.cc:5868] T 00000000000000000000000000000000 P dc3224232c314c69b9e6e86d155d8ef5: Generated new TSK 0
I20260806 09:27:15.122522 13473 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:15.141080 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 13363
W20260806 09:27:15.142475 13446 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:54143
W20260806 09:27:15.143431 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 13363: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:27:15.143682 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 13363
W20260806 09:27:15.168943 13343 connection.cc:535] client connection to 127.0.0.1:44593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:44593): Connection reset by peer (error 104)
2026-08-06T09:27:15Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2417 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:15 dist-test-slave-h3ft krb5kdc[13494](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:15 dist-test-slave-h3ft krb5kdc[13494](info): set up 2 sockets
Aug 06 09:27:15 dist-test-slave-h3ft krb5kdc[13494](info): commencing operation
krb5kdc: starting...
W20260806 09:27:16.514559 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.325s user 0.001s sys 0.064s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:16 dist-test-slave-h3ft krb5kdc[13494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008436, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:16Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:16Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:27:16.918967 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33885
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33885
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:17.358628 13514 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:17.358994 13514 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:17.359185 13514 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:17.366901 13514 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:17.367137 13514 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:17.367252 13514 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:17.367370 13514 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:17.367461 13514 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:17.378394 13514 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33885
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33885
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:17.380107 13514 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:17.381485 13514 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:17.395635 13522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:17.395744 13525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:17.395681 13521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:17.397171 13514 server_base.cc:1034] running on GCE node
I20260806 09:27:17.398437 13514 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:17.400532 13514 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:17.401890 13514 hybrid_clock.cc:648] HybridClock initialized: now 1786008437401732 us; error 69 us; skew 500 ppm
I20260806 09:27:17.402446 13514 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:17 dist-test-slave-h3ft krb5kdc[13494](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008437, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:17.406746 13514 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:27:17.408950 13514 webserver.cc:460] Webserver started at http://127.0.0.1:39669/ using document root <none> and password file <none>
I20260806 09:27:17.409963 13514 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:17.410137 13514 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:17.410552 13514 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:17.414812 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "97cf589771fe4bdea4f7f8d03202877e"
format_stamp: "Formatted at 2026-08-06 09:27:17 on dist-test-slave-h3ft"
server_key: "1fa1fabfb146a0f7e119ee8b01c7cdcc"
server_key_iv: "af7c725647b7013d73955e685f91c264"
server_key_version: "encryptionkey@0"
I20260806 09:27:17.415967 13514 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "97cf589771fe4bdea4f7f8d03202877e"
format_stamp: "Formatted at 2026-08-06 09:27:17 on dist-test-slave-h3ft"
server_key: "1fa1fabfb146a0f7e119ee8b01c7cdcc"
server_key_iv: "af7c725647b7013d73955e685f91c264"
server_key_version: "encryptionkey@0"
I20260806 09:27:17.423568 13514 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260806 09:27:17.428272 13534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:17.430192 13514 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:27:17.430466 13514 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "97cf589771fe4bdea4f7f8d03202877e"
format_stamp: "Formatted at 2026-08-06 09:27:17 on dist-test-slave-h3ft"
server_key: "1fa1fabfb146a0f7e119ee8b01c7cdcc"
server_key_iv: "af7c725647b7013d73955e685f91c264"
server_key_version: "encryptionkey@0"
I20260806 09:27:17.430709 13514 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:17.437892 13514 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:17.442986 13514 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:17.443352 13514 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:17.467300 13514 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33885
I20260806 09:27:17.467418 13600 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:33885 every 8 connection(s)
I20260806 09:27:17.469192 13514 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:17.471638 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 13514
I20260806 09:27:17.471899 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:17.472537 16730 external_mini_cluster.cc:1428] Setting key 358bd0959b6c8addcb33c4a12bede7e6
I20260806 09:27:17.474228 13603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:17.483362 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:17.474884 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56185 (local address 127.0.0.1:33885)
0806 09:27:17.476052 (+ 1168us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:17.476070 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:17.476132 (+ 62us) server_negotiation.cc:407] Connection header received
0806 09:27:17.478190 (+ 2058us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:17.478221 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:17.478798 (+ 577us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:17.479478 (+ 680us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:17.481002 (+ 1524us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:17.481009 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:17.481027 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:17.481044 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:17.481144 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:17.481171 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:17.481510 (+ 339us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:17.481705 (+ 195us) server_negotiation.cc:299] Negotiation successful
0806 09:27:17.481977 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":810,"thread_start_us":465,"threads_started":1}
I20260806 09:27:17.488507 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:17.489260 13603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e: Bootstrap starting.
I20260806 09:27:17.492754 13603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:17.493842 13603 log.cc:824] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:17.497795 13606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:17.494274 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51607 (local address 127.0.0.1:33885)
0806 09:27:17.494611 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:17.494624 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:17.495409 (+ 785us) server_negotiation.cc:407] Connection header received
0806 09:27:17.495644 (+ 235us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:17.495652 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:17.495731 (+ 79us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:17.495890 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:17.496957 (+ 1067us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:17.496963 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:17.496967 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:17.496990 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:17.497051 (+ 61us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:17.497080 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:17.497243 (+ 163us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:17.497381 (+ 138us) server_negotiation.cc:299] Negotiation successful
0806 09:27:17.497454 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":98}
I20260806 09:27:17.498351 13603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e: No bootstrap required, opened a new log
I20260806 09:27:17.507419 13603 raft_consensus.cc:348] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cf589771fe4bdea4f7f8d03202877e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33885 } }
I20260806 09:27:17.507922 13603 raft_consensus.cc:374] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:17.508050 13603 raft_consensus.cc:729] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97cf589771fe4bdea4f7f8d03202877e, State: Initialized, Role: FOLLOWER
I20260806 09:27:17.508761 13603 consensus_queue.cc:260] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cf589771fe4bdea4f7f8d03202877e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33885 } }
I20260806 09:27:17.509034 13603 raft_consensus.cc:388] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:17.509209 13603 raft_consensus.cc:482] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:17.509404 13603 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:17.512357 13603 raft_consensus.cc:504] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cf589771fe4bdea4f7f8d03202877e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33885 } }
I20260806 09:27:17.513041 13603 leader_election.cc:302] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 97cf589771fe4bdea4f7f8d03202877e; no voters:
I20260806 09:27:17.513741 13603 leader_election.cc:288] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:17.513988 13615 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:17.515992 13603 sys_catalog.cc:564] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:17.516448 13615 raft_consensus.cc:686] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [term 1 LEADER]: Becoming Leader. State: Replica: 97cf589771fe4bdea4f7f8d03202877e, State: Running, Role: LEADER
I20260806 09:27:17.517148 13615 consensus_queue.cc:237] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cf589771fe4bdea4f7f8d03202877e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33885 } }
I20260806 09:27:17.523231 13617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 97cf589771fe4bdea4f7f8d03202877e. Latest consensus state: current_term: 1 leader_uuid: "97cf589771fe4bdea4f7f8d03202877e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cf589771fe4bdea4f7f8d03202877e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33885 } } }
I20260806 09:27:17.524757 13617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:17.523147 13616 sys_catalog.cc:455] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "97cf589771fe4bdea4f7f8d03202877e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "97cf589771fe4bdea4f7f8d03202877e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33885 } } }
I20260806 09:27:17.525261 13616 sys_catalog.cc:458] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e [sys.catalog]: This master's current role is: LEADER
W20260806 09:27:17.534103 13631 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:27:17.534336 13631 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:27:17.534622 13632 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:17.544356 13632 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:17.555442 13632 catalog_manager.cc:1347] Generated new cluster ID: a949ce7ddb3e46acbb2f460359472329
I20260806 09:27:17.555615 13632 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:17.599932 13632 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:17.600595 13632 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:17.609956 13632 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 97cf589771fe4bdea4f7f8d03202877e: Generated new TSK 0
I20260806 09:27:17.611279 13632 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:17.627396 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 13514
W20260806 09:27:17.628863 13595 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:56185
W20260806 09:27:17.630187 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 13514: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:27:17.630414 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 13514
2026-08-06T09:27:17Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2491 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:17 dist-test-slave-h3ft krb5kdc[13643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:17 dist-test-slave-h3ft krb5kdc[13643](info): set up 2 sockets
Aug 06 09:27:17 dist-test-slave-h3ft krb5kdc[13643](info): commencing operation
krb5kdc: starting...
W20260806 09:27:19.010972 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.330s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:19 dist-test-slave-h3ft krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:19Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:27:19.414901 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43989
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35813
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43989
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:19.836525 13661 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:19.837024 13661 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:19.837221 13661 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:19.845328 13661 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:19.845526 13661 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:19.845646 13661 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:19.845759 13661 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:19.845849 13661 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:19.857344 13661 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35813
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43989
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43989
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:19.859162 13661 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:19.860791 13661 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:19.875437 13666 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:19.875430 13672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:19.875718 13667 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:19.876701 13661 server_base.cc:1034] running on GCE node
I20260806 09:27:19.877983 13661 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:19.880060 13661 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:19.881327 13661 hybrid_clock.cc:648] HybridClock initialized: now 1786008439881262 us; error 60 us; skew 500 ppm
I20260806 09:27:19.881788 13661 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:19 dist-test-slave-h3ft krb5kdc[13643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008439, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:19.885649 13661 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:27:19.887650 13661 webserver.cc:460] Webserver started at http://127.0.0.1:37745/ using document root <none> and password file <none>
I20260806 09:27:19.888527 13661 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:19.888689 13661 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:19.889055 13661 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:19.892738 13661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "7863c797f9d0453192e72cc37a245b90"
format_stamp: "Formatted at 2026-08-06 09:27:19 on dist-test-slave-h3ft"
server_key: "0b780db9448599bac8c5a1b7dc13836a"
server_key_iv: "51802413fa06ab7a0e540b6044a04624"
server_key_version: "encryptionkey@0"
I20260806 09:27:19.893651 13661 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "7863c797f9d0453192e72cc37a245b90"
format_stamp: "Formatted at 2026-08-06 09:27:19 on dist-test-slave-h3ft"
server_key: "0b780db9448599bac8c5a1b7dc13836a"
server_key_iv: "51802413fa06ab7a0e540b6044a04624"
server_key_version: "encryptionkey@0"
I20260806 09:27:19.900270 13661 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260806 09:27:19.905156 13683 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:19.907150 13661 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.006s sys 0.000s
I20260806 09:27:19.907445 13661 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "7863c797f9d0453192e72cc37a245b90"
format_stamp: "Formatted at 2026-08-06 09:27:19 on dist-test-slave-h3ft"
server_key: "0b780db9448599bac8c5a1b7dc13836a"
server_key_iv: "51802413fa06ab7a0e540b6044a04624"
server_key_version: "encryptionkey@0"
I20260806 09:27:19.907743 13661 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:19.936106 13661 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:19.941071 13661 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:19.941411 13661 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:19.964051 13661 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43989
I20260806 09:27:19.964160 13747 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43989 every 8 connection(s)
I20260806 09:27:19.965812 13661 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:19.970696 13749 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:19.975687 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 13661
I20260806 09:27:19.975915 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:19.976420 16730 external_mini_cluster.cc:1428] Setting key 215227936eafb390e2ef8b9df639a940
I20260806 09:27:19.985334 13749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90: Bootstrap starting.
I20260806 09:27:19.989948 13749 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:19.991564 13749 log.cc:824] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:27:19 dist-test-slave-h3ft krb5kdc[13643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008439, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 09:27:19.995587 13749 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90: No bootstrap required, opened a new log
I20260806 09:27:20.007889 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:19.978749 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39587 (local address 127.0.0.1:43989)
0806 09:27:19.979828 (+ 1079us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:19.979849 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:19.979916 (+ 67us) server_negotiation.cc:407] Connection header received
0806 09:27:19.983213 (+ 3297us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:19.983251 (+ 38us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:19.983910 (+ 659us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:19.984470 (+ 560us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:19.986014 (+ 1544us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:19.988254 (+ 2240us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:19.989641 (+ 1387us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:19.990941 (+ 1300us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:19.991209 (+ 268us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:19.996080 (+ 4871us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:19.996094 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:19.996124 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:19.996187 (+ 63us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:20.001052 (+ 4865us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:20.001980 (+ 928us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:20.001993 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:20.002005 (+ 12us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:20.002309 (+ 304us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:20.002915 (+ 606us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:20.002920 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:20.002924 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:20.004050 (+ 1126us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:20.004477 (+ 427us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:20.004989 (+ 512us) server_negotiation.cc:299] Negotiation successful
0806 09:27:20.005738 (+ 749us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":764,"thread_start_us":471,"threads_started":1}
I20260806 09:27:20.008571 13749 raft_consensus.cc:348] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7863c797f9d0453192e72cc37a245b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43989 } }
I20260806 09:27:20.009209 13749 raft_consensus.cc:374] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:20.009346 13749 raft_consensus.cc:729] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7863c797f9d0453192e72cc37a245b90, State: Initialized, Role: FOLLOWER
I20260806 09:27:20.010461 13749 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7863c797f9d0453192e72cc37a245b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43989 } }
I20260806 09:27:20.011060 13749 raft_consensus.cc:388] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:20.011282 13749 raft_consensus.cc:482] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:20.011545 13749 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:20.014137 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:20.015389 13749 raft_consensus.cc:504] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7863c797f9d0453192e72cc37a245b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43989 } }
I20260806 09:27:20.016211 13749 leader_election.cc:302] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7863c797f9d0453192e72cc37a245b90; no voters:
I20260806 09:27:20.017128 13749 leader_election.cc:288] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:20.017254 13755 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:20.021188 13755 raft_consensus.cc:686] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [term 1 LEADER]: Becoming Leader. State: Replica: 7863c797f9d0453192e72cc37a245b90, State: Running, Role: LEADER
I20260806 09:27:20.022171 13749 sys_catalog.cc:564] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:20.022102 13755 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7863c797f9d0453192e72cc37a245b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43989 } }
I20260806 09:27:20.028672 13759 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7863c797f9d0453192e72cc37a245b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7863c797f9d0453192e72cc37a245b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43989 } } }
I20260806 09:27:20.029348 13759 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:20.030159 13761 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7863c797f9d0453192e72cc37a245b90. Latest consensus state: current_term: 1 leader_uuid: "7863c797f9d0453192e72cc37a245b90" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7863c797f9d0453192e72cc37a245b90" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43989 } } }
I20260806 09:27:20.030546 13761 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:20.040277 13773 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:20.045019 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:20.022088 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45031 (local address 127.0.0.1:43989)
0806 09:27:20.022321 (+ 233us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:20.022331 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:20.022904 (+ 573us) server_negotiation.cc:407] Connection header received
0806 09:27:20.023155 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:20.023160 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:20.023241 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:20.023364 (+ 123us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:20.025457 (+ 2093us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:20.026607 (+ 1150us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:20.028540 (+ 1933us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:20.029245 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:20.029351 (+ 106us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:20.033942 (+ 4591us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:20.033953 (+ 11us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:20.033960 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:20.034025 (+ 65us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:20.040166 (+ 6141us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:20.041432 (+ 1266us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:20.041439 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:20.041444 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:20.041632 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:20.042270 (+ 638us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:20.042278 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:20.042284 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:20.043042 (+ 758us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:20.043432 (+ 390us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:20.044277 (+ 845us) server_negotiation.cc:299] Negotiation successful
0806 09:27:20.044577 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20260806 09:27:20.055343 13773 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:20.067036 13773 catalog_manager.cc:1347] Generated new cluster ID: 7b8db255ede3495db6d3b3e8f060650b
I20260806 09:27:20.067356 13773 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:20.087575 13773 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:20.089402 13773 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:20.103143 13773 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 7863c797f9d0453192e72cc37a245b90: Generated new TSK 0
I20260806 09:27:20.104496 13773 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:20.117705 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 13661
I20260806 09:27:20.119971 13742 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:20.219906 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 13661
2026-08-06T09:27:20Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/3 (2586 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:20 dist-test-slave-h3ft krb5kdc[13797](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:20 dist-test-slave-h3ft krb5kdc[13797](info): set up 2 sockets
Aug 06 09:27:20 dist-test-slave-h3ft krb5kdc[13797](info): commencing operation
krb5kdc: starting...
W20260806 09:27:21.628319 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.360s user 0.003s sys 0.061s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:21 dist-test-slave-h3ft krb5kdc[13797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008441, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:21Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:27:22.043819 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40717
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35765
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:40717
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:22.435467 13816 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:22.435850 13816 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:22.436029 13816 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:22.443531 13816 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:22.443719 13816 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:22.443833 13816 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:22.443951 13816 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:22.444032 13816 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:22.454559 13816 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35765
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:40717
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:40717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:22.456111 13816 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:22.457556 13816 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:22.471660 13824 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:22.471494 13829 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:22.471653 13825 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:22.472728 13816 server_base.cc:1034] running on GCE node
I20260806 09:27:22.473935 13816 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:22.476089 13816 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:22.477385 13816 hybrid_clock.cc:648] HybridClock initialized: now 1786008442477293 us; error 72 us; skew 500 ppm
I20260806 09:27:22.477975 13816 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:22 dist-test-slave-h3ft krb5kdc[13797](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008442, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:22.482571 13816 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:27:22.484899 13816 webserver.cc:460] Webserver started at http://127.0.0.1:46231/ using document root <none> and password file <none>
I20260806 09:27:22.485900 13816 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:22.486037 13816 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:22.486389 13816 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:22.490501 13816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "766d19effc47463585a1b7bc166f494a"
format_stamp: "Formatted at 2026-08-06 09:27:22 on dist-test-slave-h3ft"
server_key: "3e7c3c10ccd734a4ddc79d126d0e9e2b"
server_key_iv: "6f6787f216d42e45757c769d75b00901"
server_key_version: "encryptionkey@0"
I20260806 09:27:22.491632 13816 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "766d19effc47463585a1b7bc166f494a"
format_stamp: "Formatted at 2026-08-06 09:27:22 on dist-test-slave-h3ft"
server_key: "3e7c3c10ccd734a4ddc79d126d0e9e2b"
server_key_iv: "6f6787f216d42e45757c769d75b00901"
server_key_version: "encryptionkey@0"
I20260806 09:27:22.499171 13816 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260806 09:27:22.503914 13838 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:22.506176 13816 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260806 09:27:22.506431 13816 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "766d19effc47463585a1b7bc166f494a"
format_stamp: "Formatted at 2026-08-06 09:27:22 on dist-test-slave-h3ft"
server_key: "3e7c3c10ccd734a4ddc79d126d0e9e2b"
server_key_iv: "6f6787f216d42e45757c769d75b00901"
server_key_version: "encryptionkey@0"
I20260806 09:27:22.506699 13816 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:22.532730 13816 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:22.537846 13816 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:22.538218 13816 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:22.561590 13816 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:40717
I20260806 09:27:22.561655 13916 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:40717 every 8 connection(s)
I20260806 09:27:22.563494 13816 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:22.567579 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 13816
I20260806 09:27:22.567829 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:22.568490 16730 external_mini_cluster.cc:1428] Setting key 1456163ae6fd1e8ef7edb7384724b401
I20260806 09:27:22.568548 13918 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:22.586131 13918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a: Bootstrap starting.
I20260806 09:27:22.588138 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:22.572248 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45367 (local address 127.0.0.1:40717)
0806 09:27:22.573727 (+ 1479us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:22.573753 (+ 26us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:22.574131 (+ 378us) server_negotiation.cc:407] Connection header received
0806 09:27:22.576855 (+ 2724us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:22.576897 (+ 42us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:22.577488 (+ 591us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:22.577935 (+ 447us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:22.578657 (+ 722us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:22.580219 (+ 1562us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:22.582519 (+ 2300us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:22.583483 (+ 964us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:22.583634 (+ 151us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:22.585177 (+ 1543us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:22.585186 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:22.585203 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:22.585895 (+ 692us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:22.585991 (+ 96us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:22.586018 (+ 27us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:22.586117 (+ 99us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:22.586464 (+ 347us) server_negotiation.cc:299] Negotiation successful
0806 09:27:22.586814 (+ 350us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1096,"thread_start_us":713,"threads_started":1}
I20260806 09:27:22.590169 13918 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:22.591626 13918 log.cc:824] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:22.592885 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:22.594867 13918 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a: No bootstrap required, opened a new log
I20260806 09:27:22.604703 13918 raft_consensus.cc:348] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766d19effc47463585a1b7bc166f494a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } }
I20260806 09:27:22.605278 13918 raft_consensus.cc:374] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:22.605378 13922 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:22.598097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34419 (local address 127.0.0.1:40717)
0806 09:27:22.598294 (+ 197us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:22.598300 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:22.598942 (+ 642us) server_negotiation.cc:407] Connection header received
0806 09:27:22.599260 (+ 318us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:22.599265 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:22.599338 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:22.599471 (+ 133us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:22.600106 (+ 635us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:22.600931 (+ 825us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:22.602339 (+ 1408us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:22.602899 (+ 560us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:22.603093 (+ 194us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:22.604449 (+ 1356us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:22.604454 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:22.604458 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:22.604481 (+ 23us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:22.604527 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:22.604545 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:22.604652 (+ 107us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:22.604807 (+ 155us) server_negotiation.cc:299] Negotiation successful
0806 09:27:22.604932 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:27:22.605485 13918 raft_consensus.cc:729] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 766d19effc47463585a1b7bc166f494a, State: Initialized, Role: FOLLOWER
I20260806 09:27:22.606288 13918 consensus_queue.cc:260] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766d19effc47463585a1b7bc166f494a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } }
I20260806 09:27:22.606637 13918 raft_consensus.cc:388] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:22.606822 13918 raft_consensus.cc:482] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:22.607120 13918 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:22.610275 13918 raft_consensus.cc:504] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766d19effc47463585a1b7bc166f494a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } }
I20260806 09:27:22.610860 13918 leader_election.cc:302] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 766d19effc47463585a1b7bc166f494a; no voters:
I20260806 09:27:22.611649 13918 leader_election.cc:288] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:22.611923 13931 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:22.612989 13931 raft_consensus.cc:686] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [term 1 LEADER]: Becoming Leader. State: Replica: 766d19effc47463585a1b7bc166f494a, State: Running, Role: LEADER
I20260806 09:27:22.613747 13931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766d19effc47463585a1b7bc166f494a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } }
I20260806 09:27:22.614455 13918 sys_catalog.cc:564] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:22.617244 13932 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "766d19effc47463585a1b7bc166f494a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766d19effc47463585a1b7bc166f494a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } } }
I20260806 09:27:22.617702 13932 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:22.618813 13933 sys_catalog.cc:455] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 766d19effc47463585a1b7bc166f494a. Latest consensus state: current_term: 1 leader_uuid: "766d19effc47463585a1b7bc166f494a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "766d19effc47463585a1b7bc166f494a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 40717 } } }
I20260806 09:27:22.619800 13933 sys_catalog.cc:458] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:22.624017 13936 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:22.630543 13936 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:22.641501 13936 catalog_manager.cc:1347] Generated new cluster ID: 9fc11c95d30749eaa2025b44dd4503ae
I20260806 09:27:22.641667 13936 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:22.677627 13936 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:22.679951 13936 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:22.693948 13936 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 766d19effc47463585a1b7bc166f494a: Generated new TSK 0
I20260806 09:27:22.695312 13936 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:22.703830 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 13816
W20260806 09:27:22.705237 13910 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:45367
W20260806 09:27:22.706529 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 13816: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:27:22.706918 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 13816
2026-08-06T09:27:22Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2488 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:22 dist-test-slave-h3ft krb5kdc[13957](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:22 dist-test-slave-h3ft krb5kdc[13957](info): set up 2 sockets
Aug 06 09:27:22 dist-test-slave-h3ft krb5kdc[13957](info): commencing operation
krb5kdc: starting...
W20260806 09:27:24.114967 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.358s user 0.000s sys 0.057s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:24 dist-test-slave-h3ft krb5kdc[13957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008444, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:24Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:24Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:27:24.529251 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36343
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:45133
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36343
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:24.963724 13976 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:24.964185 13976 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:24.964300 13976 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:24.972872 13976 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:24.973057 13976 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:24.973119 13976 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:24.973201 13976 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:24.973259 13976 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:24.984999 13976 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45133
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36343
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36343
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:24.986753 13976 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:24.988701 13976 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:25.004263 13984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:25.004287 13986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:25.004650 13989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:25.005750 13976 server_base.cc:1034] running on GCE node
I20260806 09:27:25.006853 13976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:25.008867 13976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:25.010171 13976 hybrid_clock.cc:648] HybridClock initialized: now 1786008445010116 us; error 41 us; skew 500 ppm
I20260806 09:27:25.010645 13976 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:25 dist-test-slave-h3ft krb5kdc[13957](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008445, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:25.019526 13976 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:27:25.021601 13976 webserver.cc:460] Webserver started at http://127.0.0.1:38335/ using document root <none> and password file <none>
I20260806 09:27:25.022517 13976 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:25.022689 13976 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:25.023130 13976 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:25.026798 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "2be206f863374a06ae8a766256cf3ead"
format_stamp: "Formatted at 2026-08-06 09:27:25 on dist-test-slave-h3ft"
server_key: "e8b6117a56d9e45a956da95ec0c9051d"
server_key_iv: "e00cc652053db39e5e08624033d9535a"
server_key_version: "encryptionkey@0"
I20260806 09:27:25.027777 13976 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "2be206f863374a06ae8a766256cf3ead"
format_stamp: "Formatted at 2026-08-06 09:27:25 on dist-test-slave-h3ft"
server_key: "e8b6117a56d9e45a956da95ec0c9051d"
server_key_iv: "e00cc652053db39e5e08624033d9535a"
server_key_version: "encryptionkey@0"
I20260806 09:27:25.034265 13976 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20260806 09:27:25.038652 14004 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:25.040652 13976 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:27:25.040926 13976 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "2be206f863374a06ae8a766256cf3ead"
format_stamp: "Formatted at 2026-08-06 09:27:25 on dist-test-slave-h3ft"
server_key: "e8b6117a56d9e45a956da95ec0c9051d"
server_key_iv: "e00cc652053db39e5e08624033d9535a"
server_key_version: "encryptionkey@0"
I20260806 09:27:25.041219 13976 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:25.048141 13976 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:25.053013 13976 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:25.053367 13976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:25.077054 13976 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36343
I20260806 09:27:25.077168 14078 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:36343 every 8 connection(s)
I20260806 09:27:25.078939 13976 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:25.082772 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 13976
I20260806 09:27:25.083091 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:25.083653 16730 external_mini_cluster.cc:1428] Setting key c29c3b507cf3ce70bf478374eae32f37
I20260806 09:27:25.083884 14079 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:25.094161 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:25.086000 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36913 (local address 127.0.0.1:36343)
0806 09:27:25.087150 (+ 1150us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:25.087165 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:25.087227 (+ 62us) server_negotiation.cc:407] Connection header received
0806 09:27:25.089329 (+ 2102us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:25.089359 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:25.089850 (+ 491us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:25.090320 (+ 470us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:25.091900 (+ 1580us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:25.091912 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:25.091938 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:25.091979 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:25.092077 (+ 98us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:25.092111 (+ 34us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:25.092390 (+ 279us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:25.092565 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 09:27:25.092921 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":752,"thread_start_us":469,"threads_started":1}
I20260806 09:27:25.099831 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:25.100013 14079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead: Bootstrap starting.
I20260806 09:27:25.103403 14079 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:25.104727 14079 log.cc:824] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:25.109047 14079 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead: No bootstrap required, opened a new log
I20260806 09:27:25.109138 14083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:25.105774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55727 (local address 127.0.0.1:36343)
0806 09:27:25.106034 (+ 260us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:25.106044 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:25.106476 (+ 432us) server_negotiation.cc:407] Connection header received
0806 09:27:25.106774 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:25.106779 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:25.106847 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:25.106958 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:25.108213 (+ 1255us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:25.108218 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:25.108222 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:25.108239 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:25.108288 (+ 49us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:25.108306 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:25.108446 (+ 140us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:25.108624 (+ 178us) server_negotiation.cc:299] Negotiation successful
0806 09:27:25.108687 (+ 63us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20260806 09:27:25.117254 14079 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be206f863374a06ae8a766256cf3ead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36343 } }
I20260806 09:27:25.117667 14079 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:25.117833 14079 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2be206f863374a06ae8a766256cf3ead, State: Initialized, Role: FOLLOWER
I20260806 09:27:25.118436 14079 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be206f863374a06ae8a766256cf3ead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36343 } }
I20260806 09:27:25.118697 14079 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:25.118906 14079 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:25.119136 14079 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:25.121858 14079 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be206f863374a06ae8a766256cf3ead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36343 } }
I20260806 09:27:25.122362 14079 leader_election.cc:302] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2be206f863374a06ae8a766256cf3ead; no voters:
I20260806 09:27:25.123230 14079 leader_election.cc:288] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:25.123526 14095 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:25.124606 14095 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [term 1 LEADER]: Becoming Leader. State: Replica: 2be206f863374a06ae8a766256cf3ead, State: Running, Role: LEADER
I20260806 09:27:25.125344 14095 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be206f863374a06ae8a766256cf3ead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36343 } }
I20260806 09:27:25.125674 14079 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:25.131889 14096 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2be206f863374a06ae8a766256cf3ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be206f863374a06ae8a766256cf3ead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36343 } } }
I20260806 09:27:25.132424 14096 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:25.133255 14100 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2be206f863374a06ae8a766256cf3ead. Latest consensus state: current_term: 1 leader_uuid: "2be206f863374a06ae8a766256cf3ead" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2be206f863374a06ae8a766256cf3ead" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36343 } } }
I20260806 09:27:25.133677 14100 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:25.135666 14117 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:25.142741 14117 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:25.153867 14117 catalog_manager.cc:1347] Generated new cluster ID: 207fa71002ac441dbdaee67be84097c5
I20260806 09:27:25.154021 14117 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:25.167565 14117 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:25.168188 14117 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:25.180109 14117 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2be206f863374a06ae8a766256cf3ead: Generated new TSK 0
I20260806 09:27:25.181401 14117 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:25.194595 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 13976
W20260806 09:27:25.196522 14072 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:36913
W20260806 09:27:25.197540 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 13976: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:27:25.197732 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 13976
2026-08-06T09:27:25Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2489 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:25 dist-test-slave-h3ft krb5kdc[14128](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:25 dist-test-slave-h3ft krb5kdc[14128](info): set up 2 sockets
Aug 06 09:27:25 dist-test-slave-h3ft krb5kdc[14128](info): commencing operation
krb5kdc: starting...
W20260806 09:27:26.582116 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.334s user 0.002s sys 0.066s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:26 dist-test-slave-h3ft krb5kdc[14128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:26Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:27:27.000036 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:37437
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:27.407192 14144 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:27.407521 14144 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:27.407616 14144 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:27.415174 14144 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:27.415315 14144 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:27.415366 14144 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:27.415422 14144 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:27.415462 14144 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:27.425647 14144 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:37437
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43781
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:27.426968 14144 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:27.428310 14144 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:27.441640 14151 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:27.441710 14152 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:27.442327 14156 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:27.443941 14144 server_base.cc:1034] running on GCE node
I20260806 09:27:27.444821 14144 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:27.446350 14144 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:27.447589 14144 hybrid_clock.cc:648] HybridClock initialized: now 1786008447447493 us; error 70 us; skew 500 ppm
I20260806 09:27:27.447956 14144 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:27 dist-test-slave-h3ft krb5kdc[14128](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008447, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:27.451439 14144 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:27:27.453295 14144 webserver.cc:460] Webserver started at http://127.0.0.1:39641/ using document root <none> and password file <none>
I20260806 09:27:27.454028 14144 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:27.454191 14144 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:27.454519 14144 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:27.457700 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "314f6b5c16e64972a38dec63c9c85345"
format_stamp: "Formatted at 2026-08-06 09:27:27 on dist-test-slave-h3ft"
server_key: "34ef5a5732f8904f1911e33090e30ddd"
server_key_iv: "17fd34113a5832a395fbf0e993e047ef"
server_key_version: "encryptionkey@0"
I20260806 09:27:27.458518 14144 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "314f6b5c16e64972a38dec63c9c85345"
format_stamp: "Formatted at 2026-08-06 09:27:27 on dist-test-slave-h3ft"
server_key: "34ef5a5732f8904f1911e33090e30ddd"
server_key_iv: "17fd34113a5832a395fbf0e993e047ef"
server_key_version: "encryptionkey@0"
I20260806 09:27:27.464587 14144 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 09:27:27.468971 14163 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:27.470682 14144 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:27:27.470911 14144 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "314f6b5c16e64972a38dec63c9c85345"
format_stamp: "Formatted at 2026-08-06 09:27:27 on dist-test-slave-h3ft"
server_key: "34ef5a5732f8904f1911e33090e30ddd"
server_key_iv: "17fd34113a5832a395fbf0e993e047ef"
server_key_version: "encryptionkey@0"
I20260806 09:27:27.471191 14144 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:27.534134 14144 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:27.538821 14144 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:27.539238 14144 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:27.561718 14144 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43781
I20260806 09:27:27.561779 14242 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43781 every 8 connection(s)
I20260806 09:27:27.563271 14144 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:27.565730 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 14144
I20260806 09:27:27.565936 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:27.566421 16730 external_mini_cluster.cc:1428] Setting key 1ec5707d18d2ba65333bc91abac927f7
I20260806 09:27:27.568964 14243 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:27:27 dist-test-slave-h3ft krb5kdc[14128](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008446, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 09:27:27.582861 14243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345: Bootstrap starting.
I20260806 09:27:27.587352 14243 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:27.588665 14243 log.cc:824] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:27.592607 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:27.568683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:36057 (local address 127.0.0.1:43781)
0806 09:27:27.569725 (+ 1042us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:27.569742 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:27.569792 (+ 50us) server_negotiation.cc:407] Connection header received
0806 09:27:27.571708 (+ 1916us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:27.571736 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:27.572244 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:27.572741 (+ 497us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:27.574386 (+ 1645us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:27.576327 (+ 1941us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:27.577616 (+ 1289us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:27.578362 (+ 746us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:27.578567 (+ 205us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:27.583286 (+ 4719us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:27.583295 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:27.583313 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:27.583363 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:27.586788 (+ 3425us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:27.587864 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:27.587872 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:27.587879 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:27.588067 (+ 188us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:27.588575 (+ 508us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:27.588579 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:27.588581 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:27.589265 (+ 684us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:27.589662 (+ 397us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:27.590394 (+ 732us) server_negotiation.cc:299] Negotiation successful
0806 09:27:27.590804 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":767,"thread_start_us":478,"threads_started":1}
I20260806 09:27:27.593191 14243 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345: No bootstrap required, opened a new log
I20260806 09:27:27.598585 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:27.605579 14243 raft_consensus.cc:348] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314f6b5c16e64972a38dec63c9c85345" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260806 09:27:27.606222 14243 raft_consensus.cc:374] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:27.606384 14243 raft_consensus.cc:729] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 314f6b5c16e64972a38dec63c9c85345, State: Initialized, Role: FOLLOWER
I20260806 09:27:27.607239 14243 consensus_queue.cc:260] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314f6b5c16e64972a38dec63c9c85345" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260806 09:27:27.607553 14243 raft_consensus.cc:388] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:27.607735 14243 raft_consensus.cc:482] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:27.607963 14243 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:27.610714 14243 raft_consensus.cc:504] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314f6b5c16e64972a38dec63c9c85345" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260806 09:27:27.611481 14243 leader_election.cc:302] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 314f6b5c16e64972a38dec63c9c85345; no voters:
I20260806 09:27:27.612347 14243 leader_election.cc:288] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:27.612571 14254 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:27.615041 14243 sys_catalog.cc:564] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:27.617622 14254 raft_consensus.cc:686] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [term 1 LEADER]: Becoming Leader. State: Replica: 314f6b5c16e64972a38dec63c9c85345, State: Running, Role: LEADER
I20260806 09:27:27.618505 14254 consensus_queue.cc:237] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314f6b5c16e64972a38dec63c9c85345" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } }
I20260806 09:27:27.621007 14254 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 314f6b5c16e64972a38dec63c9c85345. Latest consensus state: current_term: 1 leader_uuid: "314f6b5c16e64972a38dec63c9c85345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314f6b5c16e64972a38dec63c9c85345" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } } }
I20260806 09:27:27.621421 14254 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:27.621932 14255 sys_catalog.cc:455] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "314f6b5c16e64972a38dec63c9c85345" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "314f6b5c16e64972a38dec63c9c85345" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43781 } } }
I20260806 09:27:27.622291 14255 sys_catalog.cc:458] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:27.627674 14246 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:27.605042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:54847 (local address 127.0.0.1:43781)
0806 09:27:27.605335 (+ 293us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:27.605345 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:27.605937 (+ 592us) server_negotiation.cc:407] Connection header received
0806 09:27:27.606215 (+ 278us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:27.606219 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:27.606297 (+ 78us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:27.606491 (+ 194us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:27.608277 (+ 1786us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:27.609158 (+ 881us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:27.610431 (+ 1273us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:27.610964 (+ 533us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:27.611132 (+ 168us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:27.615386 (+ 4254us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:27.615396 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:27.615404 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:27.615466 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:27.621406 (+ 5940us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:27.623118 (+ 1712us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:27.623129 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:27.623136 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:27.623418 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:27.624494 (+ 1076us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:27.624503 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:27.624509 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:27.625201 (+ 692us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:27.626709 (+ 1508us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:27.626923 (+ 214us) server_negotiation.cc:299] Negotiation successful
0806 09:27:27.627218 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20260806 09:27:27.629820 14268 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:27.636736 14268 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:27.648195 14268 catalog_manager.cc:1347] Generated new cluster ID: 00bab6411a3b4767a844a42b72e3e431
I20260806 09:27:27.648392 14268 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:27.661221 14268 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:27.663467 14268 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:27.675971 14268 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 314f6b5c16e64972a38dec63c9c85345: Generated new TSK 0
I20260806 09:27:27.677309 14268 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:27.691908 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 14144
I20260806 09:27:27.694226 14236 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:27.794293 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 14144
W20260806 09:27:27.826704 14125 connection.cc:535] client connection to 127.0.0.1:43781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43781): Connection reset by peer (error 104)
2026-08-06T09:27:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2601 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:27 dist-test-slave-h3ft krb5kdc[14285](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:27 dist-test-slave-h3ft krb5kdc[14285](info): set up 2 sockets
Aug 06 09:27:27 dist-test-slave-h3ft krb5kdc[14285](info): commencing operation
krb5kdc: starting...
W20260806 09:27:29.154737 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.306s user 0.000s sys 0.054s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:29 dist-test-slave-h3ft krb5kdc[14285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008449, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:29Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:27:29.547674 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39725
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:41659
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39725
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:29.979389 14302 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:29.979800 14302 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:29.979907 14302 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:29.988101 14302 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:29.988256 14302 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:29.988317 14302 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:29.988386 14302 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:29.988437 14302 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:30.000097 14302 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41659
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39725
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39725
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:30.002004 14302 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:30.003955 14302 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:30.020951 14315 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:30.021046 14314 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:30.021029 14318 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:30.022080 14302 server_base.cc:1034] running on GCE node
I20260806 09:27:30.023198 14302 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:30.025368 14302 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:30.026643 14302 hybrid_clock.cc:648] HybridClock initialized: now 1786008450026560 us; error 81 us; skew 500 ppm
I20260806 09:27:30.027179 14302 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:30 dist-test-slave-h3ft krb5kdc[14285](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008450, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:30.036772 14302 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:27:30.038964 14302 webserver.cc:460] Webserver started at http://127.0.0.1:41629/ using document root <none> and password file <none>
I20260806 09:27:30.039956 14302 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:30.040148 14302 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:30.040553 14302 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:30.044560 14302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "a4cfb33e87344abc8500035f2bb9f7c5"
format_stamp: "Formatted at 2026-08-06 09:27:30 on dist-test-slave-h3ft"
server_key: "a2b47a8789fa461460c1e15f63c56667"
server_key_iv: "979a34a619e101364f384d4e0e885a46"
server_key_version: "encryptionkey@0"
I20260806 09:27:30.045552 14302 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "a4cfb33e87344abc8500035f2bb9f7c5"
format_stamp: "Formatted at 2026-08-06 09:27:30 on dist-test-slave-h3ft"
server_key: "a2b47a8789fa461460c1e15f63c56667"
server_key_iv: "979a34a619e101364f384d4e0e885a46"
server_key_version: "encryptionkey@0"
I20260806 09:27:30.054113 14302 fs_manager.cc:691] Time spent creating directory manager: real 0.008s user 0.007s sys 0.003s
I20260806 09:27:30.059314 14326 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:30.061264 14302 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:27:30.061563 14302 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "a4cfb33e87344abc8500035f2bb9f7c5"
format_stamp: "Formatted at 2026-08-06 09:27:30 on dist-test-slave-h3ft"
server_key: "a2b47a8789fa461460c1e15f63c56667"
server_key_iv: "979a34a619e101364f384d4e0e885a46"
server_key_version: "encryptionkey@0"
I20260806 09:27:30.061857 14302 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:30.103544 14302 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:30.109726 14302 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:30.110159 14302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:30.134992 14302 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39725
I20260806 09:27:30.135084 14394 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:39725 every 8 connection(s)
I20260806 09:27:30.137027 14302 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:30.139865 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 14302
I20260806 09:27:30.140115 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:30.140656 16730 external_mini_cluster.cc:1428] Setting key 889e50ada3d06c3e4aebcb7549ef4c4d
I20260806 09:27:30.141875 14395 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:30.157238 14395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5: Bootstrap starting.
I20260806 09:27:30.158363 14398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:30.143390 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:44149 (local address 127.0.0.1:39725)
0806 09:27:30.144485 (+ 1095us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:30.144500 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:30.144553 (+ 53us) server_negotiation.cc:407] Connection header received
0806 09:27:30.147276 (+ 2723us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:30.147307 (+ 31us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:30.147893 (+ 586us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:30.148343 (+ 450us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:30.149152 (+ 809us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:30.150729 (+ 1577us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:30.152698 (+ 1969us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:30.153586 (+ 888us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:30.153735 (+ 149us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:30.155308 (+ 1573us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:30.155317 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:30.155334 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:30.156055 (+ 721us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:30.156158 (+ 103us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:30.156188 (+ 30us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:30.156323 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:30.156688 (+ 365us) server_negotiation.cc:299] Negotiation successful
0806 09:27:30.156987 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":808,"thread_start_us":494,"threads_started":1}
I20260806 09:27:30.161223 14395 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:30.162671 14395 log.cc:824] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:30.164156 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:30.166409 14395 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5: No bootstrap required, opened a new log
I20260806 09:27:30.175668 14395 raft_consensus.cc:348] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cfb33e87344abc8500035f2bb9f7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39725 } }
I20260806 09:27:30.176102 14395 raft_consensus.cc:374] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:30.176266 14395 raft_consensus.cc:729] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a4cfb33e87344abc8500035f2bb9f7c5, State: Initialized, Role: FOLLOWER
I20260806 09:27:30.176962 14395 consensus_queue.cc:260] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cfb33e87344abc8500035f2bb9f7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39725 } }
I20260806 09:27:30.177223 14395 raft_consensus.cc:388] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:30.177407 14395 raft_consensus.cc:482] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:30.177608 14395 raft_consensus.cc:3037] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:30.178525 14398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:30.170479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:46271 (local address 127.0.0.1:39725)
0806 09:27:30.170767 (+ 288us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:30.170776 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:30.171082 (+ 306us) server_negotiation.cc:407] Connection header received
0806 09:27:30.171409 (+ 327us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:30.171414 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:30.171498 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:30.171640 (+ 142us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:30.172343 (+ 703us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:30.173376 (+ 1033us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:30.174970 (+ 1594us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:30.175572 (+ 602us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:30.175744 (+ 172us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:30.177510 (+ 1766us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:30.177519 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:30.177525 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:30.177559 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:30.177641 (+ 82us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:30.177670 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:30.177865 (+ 195us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:30.178078 (+ 213us) server_negotiation.cc:299] Negotiation successful
0806 09:27:30.178155 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 09:27:30.180948 14395 raft_consensus.cc:504] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cfb33e87344abc8500035f2bb9f7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39725 } }
I20260806 09:27:30.181510 14395 leader_election.cc:302] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a4cfb33e87344abc8500035f2bb9f7c5; no voters:
I20260806 09:27:30.182276 14395 leader_election.cc:288] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:30.182637 14408 raft_consensus.cc:2781] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:30.184249 14408 raft_consensus.cc:686] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [term 1 LEADER]: Becoming Leader. State: Replica: a4cfb33e87344abc8500035f2bb9f7c5, State: Running, Role: LEADER
I20260806 09:27:30.184911 14395 sys_catalog.cc:564] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:30.185062 14408 consensus_queue.cc:237] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cfb33e87344abc8500035f2bb9f7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39725 } }
I20260806 09:27:30.189530 14411 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a4cfb33e87344abc8500035f2bb9f7c5. Latest consensus state: current_term: 1 leader_uuid: "a4cfb33e87344abc8500035f2bb9f7c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cfb33e87344abc8500035f2bb9f7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39725 } } }
I20260806 09:27:30.190092 14411 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:30.189502 14410 sys_catalog.cc:455] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a4cfb33e87344abc8500035f2bb9f7c5" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a4cfb33e87344abc8500035f2bb9f7c5" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39725 } } }
I20260806 09:27:30.190630 14410 sys_catalog.cc:458] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:30.191720 14418 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:30.199316 14418 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:30.212220 14418 catalog_manager.cc:1347] Generated new cluster ID: 04c39b7518164699ad4c27969fdd9db1
I20260806 09:27:30.212384 14418 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:30.222647 14418 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:30.224098 14418 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:30.236027 14418 catalog_manager.cc:5868] T 00000000000000000000000000000000 P a4cfb33e87344abc8500035f2bb9f7c5: Generated new TSK 0
I20260806 09:27:30.236907 14418 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:30.245394 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 14302
W20260806 09:27:30.246850 14390 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.166:44149
W20260806 09:27:30.247813 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 14302: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:27:30.247977 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 14302
W20260806 09:27:30.277047 14282 connection.cc:535] client connection to 127.0.0.1:39725 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39725): Connection reset by peer (error 104)
2026-08-06T09:27:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2449 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:30 dist-test-slave-h3ft krb5kdc[14432](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:30 dist-test-slave-h3ft krb5kdc[14432](info): set up 2 sockets
Aug 06 09:27:30 dist-test-slave-h3ft krb5kdc[14432](info): commencing operation
krb5kdc: starting...
W20260806 09:27:31.686937 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.388s user 0.001s sys 0.068s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:31 dist-test-slave-h3ft krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008451, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:31Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:31Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:27:32.099503 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38797
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:44763
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38797
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:32.528934 14457 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:32.529449 14457 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:32.529561 14457 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:32.537621 14457 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:32.537788 14457 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:32.537842 14457 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:32.537910 14457 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:32.537962 14457 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:32.549257 14457 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:44763
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38797
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:32.551071 14457 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:32.552702 14457 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:32.567500 14465 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:32.567539 14469 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:32.568388 14464 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:32.569082 14457 server_base.cc:1034] running on GCE node
I20260806 09:27:32.570206 14457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:32.572073 14457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:32.573374 14457 hybrid_clock.cc:648] HybridClock initialized: now 1786008452573298 us; error 54 us; skew 500 ppm
I20260806 09:27:32.573817 14457 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:32 dist-test-slave-h3ft krb5kdc[14432](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008452, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:32.577665 14457 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:27:32.579572 14457 webserver.cc:460] Webserver started at http://127.0.0.1:43757/ using document root <none> and password file <none>
I20260806 09:27:32.580430 14457 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:32.580600 14457 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:32.580952 14457 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:32.584329 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "b8848fa0b2a44f25823930e60f1188c7"
format_stamp: "Formatted at 2026-08-06 09:27:32 on dist-test-slave-h3ft"
server_key: "12d7b0711d17685f975b56aa1b0e17c4"
server_key_iv: "a059f31e5c626a4b313ff4b3f517b1bc"
server_key_version: "encryptionkey@0"
I20260806 09:27:32.585212 14457 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "b8848fa0b2a44f25823930e60f1188c7"
format_stamp: "Formatted at 2026-08-06 09:27:32 on dist-test-slave-h3ft"
server_key: "12d7b0711d17685f975b56aa1b0e17c4"
server_key_iv: "a059f31e5c626a4b313ff4b3f517b1bc"
server_key_version: "encryptionkey@0"
I20260806 09:27:32.591485 14457 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260806 09:27:32.595893 14478 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:32.597615 14457 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:27:32.597842 14457 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "b8848fa0b2a44f25823930e60f1188c7"
format_stamp: "Formatted at 2026-08-06 09:27:32 on dist-test-slave-h3ft"
server_key: "12d7b0711d17685f975b56aa1b0e17c4"
server_key_iv: "a059f31e5c626a4b313ff4b3f517b1bc"
server_key_version: "encryptionkey@0"
I20260806 09:27:32.598066 14457 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:32.605049 14457 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:32.609963 14457 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:32.610314 14457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:32.633941 14457 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38797
I20260806 09:27:32.634047 14544 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:38797 every 8 connection(s)
I20260806 09:27:32.635671 14457 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:32.640663 14545 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:32.641880 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 14457
I20260806 09:27:32.642089 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:32.642632 16730 external_mini_cluster.cc:1428] Setting key 38fd9a5b373d4275bd717c8031243dee
I20260806 09:27:32.652628 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:32.645216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:39063 (local address 127.0.0.1:38797)
0806 09:27:32.646304 (+ 1088us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:32.646318 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:32.646365 (+ 47us) server_negotiation.cc:407] Connection header received
0806 09:27:32.648459 (+ 2094us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:32.648489 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:32.648891 (+ 402us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:32.649275 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:32.650777 (+ 1502us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:32.650784 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:32.650799 (+ 15us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:32.650815 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:32.650890 (+ 75us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:32.650911 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:32.651176 (+ 265us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:32.651296 (+ 120us) server_negotiation.cc:299] Negotiation successful
0806 09:27:32.651545 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":744,"thread_start_us":460,"threads_started":1}
I20260806 09:27:32.657069 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:32.657678 14545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7: Bootstrap starting.
I20260806 09:27:32.662115 14545 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:32.663925 14545 log.cc:824] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:32.666034 14548 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:32.662553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:33475 (local address 127.0.0.1:38797)
0806 09:27:32.662812 (+ 259us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:32.662825 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:32.663446 (+ 621us) server_negotiation.cc:407] Connection header received
0806 09:27:32.663686 (+ 240us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:32.663690 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:32.663749 (+ 59us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:32.663866 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:32.665119 (+ 1253us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:32.665128 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:32.665135 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:32.665157 (+ 22us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:32.665212 (+ 55us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:32.665233 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:32.665390 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:32.665612 (+ 222us) server_negotiation.cc:299] Negotiation successful
0806 09:27:32.665687 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":114}
I20260806 09:27:32.668543 14545 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7: No bootstrap required, opened a new log
I20260806 09:27:32.678506 14545 raft_consensus.cc:348] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8848fa0b2a44f25823930e60f1188c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38797 } }
I20260806 09:27:32.679133 14545 raft_consensus.cc:374] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:32.679267 14545 raft_consensus.cc:729] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b8848fa0b2a44f25823930e60f1188c7, State: Initialized, Role: FOLLOWER
I20260806 09:27:32.680421 14545 consensus_queue.cc:260] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8848fa0b2a44f25823930e60f1188c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38797 } }
I20260806 09:27:32.680763 14545 raft_consensus.cc:388] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:32.680958 14545 raft_consensus.cc:482] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:32.681216 14545 raft_consensus.cc:3037] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:32.684770 14545 raft_consensus.cc:504] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8848fa0b2a44f25823930e60f1188c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38797 } }
I20260806 09:27:32.685499 14545 leader_election.cc:302] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b8848fa0b2a44f25823930e60f1188c7; no voters:
I20260806 09:27:32.686282 14545 leader_election.cc:288] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:32.686534 14558 raft_consensus.cc:2781] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:32.687481 14558 raft_consensus.cc:686] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [term 1 LEADER]: Becoming Leader. State: Replica: b8848fa0b2a44f25823930e60f1188c7, State: Running, Role: LEADER
I20260806 09:27:32.688222 14558 consensus_queue.cc:237] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8848fa0b2a44f25823930e60f1188c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38797 } }
I20260806 09:27:32.689462 14545 sys_catalog.cc:564] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:32.691291 14560 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b8848fa0b2a44f25823930e60f1188c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8848fa0b2a44f25823930e60f1188c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38797 } } }
I20260806 09:27:32.691888 14560 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:32.692651 14561 sys_catalog.cc:455] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader b8848fa0b2a44f25823930e60f1188c7. Latest consensus state: current_term: 1 leader_uuid: "b8848fa0b2a44f25823930e60f1188c7" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b8848fa0b2a44f25823930e60f1188c7" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38797 } } }
I20260806 09:27:32.693142 14561 sys_catalog.cc:458] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:32.695551 14564 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:32.702972 14564 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:32.714581 14564 catalog_manager.cc:1347] Generated new cluster ID: 1907bf62baa94ca1ba60a0c8f22cf196
I20260806 09:27:32.714728 14564 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:32.741387 14564 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:32.741931 14564 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:32.752882 14564 catalog_manager.cc:5868] T 00000000000000000000000000000000 P b8848fa0b2a44f25823930e60f1188c7: Generated new TSK 0
I20260806 09:27:32.754021 14564 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:32.770273 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 14457
W20260806 09:27:32.772153 14539 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.166:39063
W20260806 09:27:32.773033 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 14457: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:27:32.773216 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 14457
2026-08-06T09:27:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2522 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:32 dist-test-slave-h3ft krb5kdc[14582](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:32 dist-test-slave-h3ft krb5kdc[14582](info): set up 2 sockets
Aug 06 09:27:32 dist-test-slave-h3ft krb5kdc[14582](info): commencing operation
krb5kdc: starting...
W20260806 09:27:34.157512 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.335s user 0.000s sys 0.069s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:34 dist-test-slave-h3ft krb5kdc[14582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:34Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:27:34.571751 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41381
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:35277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:41381
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:34.959197 14602 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:34.959537 14602 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:34.959713 14602 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:34.967566 14602 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:34.967713 14602 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:34.967790 14602 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:34.967916 14602 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:34.967990 14602 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:34.978219 14602 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:41381
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:41381
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:34.979667 14602 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:34.981050 14602 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:34.996006 14612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:34.995957 14615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:34.996025 14613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:34.997640 14602 server_base.cc:1034] running on GCE node
I20260806 09:27:34.998684 14602 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:35.000419 14602 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:35.001731 14602 hybrid_clock.cc:648] HybridClock initialized: now 1786008455001664 us; error 54 us; skew 500 ppm
I20260806 09:27:35.002207 14602 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:34 dist-test-slave-h3ft krb5kdc[14582](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008454, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:35.006603 14602 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:27:35.008782 14602 webserver.cc:460] Webserver started at http://127.0.0.1:44615/ using document root <none> and password file <none>
I20260806 09:27:35.009583 14602 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:35.009763 14602 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:35.010114 14602 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:35.013824 14602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "ec9e5bd27018422784a0017370ca3f61"
format_stamp: "Formatted at 2026-08-06 09:27:35 on dist-test-slave-h3ft"
server_key: "aed3d35f9d4c29a284a2abd28dbb6f35"
server_key_iv: "3a26b8c40a9ac225537297556a95f23d"
server_key_version: "encryptionkey@0"
I20260806 09:27:35.014698 14602 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "ec9e5bd27018422784a0017370ca3f61"
format_stamp: "Formatted at 2026-08-06 09:27:35 on dist-test-slave-h3ft"
server_key: "aed3d35f9d4c29a284a2abd28dbb6f35"
server_key_iv: "3a26b8c40a9ac225537297556a95f23d"
server_key_version: "encryptionkey@0"
I20260806 09:27:35.020984 14602 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.000s sys 0.004s
I20260806 09:27:35.025326 14624 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:35.026911 14602 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.001s sys 0.000s
I20260806 09:27:35.027180 14602 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "ec9e5bd27018422784a0017370ca3f61"
format_stamp: "Formatted at 2026-08-06 09:27:35 on dist-test-slave-h3ft"
server_key: "aed3d35f9d4c29a284a2abd28dbb6f35"
server_key_iv: "3a26b8c40a9ac225537297556a95f23d"
server_key_version: "encryptionkey@0"
I20260806 09:27:35.027415 14602 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:35.073398 14602 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:35.078577 14602 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:35.078962 14602 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:35.103077 14602 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:41381
I20260806 09:27:35.103258 14701 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:41381 every 8 connection(s)
I20260806 09:27:35.104655 14602 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:35.105306 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 14602
I20260806 09:27:35.105547 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:35.106303 16730 external_mini_cluster.cc:1428] Setting key 84f9f975b7660388ae8881f8a791451f
I20260806 09:27:35.110991 14702 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:27:35 dist-test-slave-h3ft krb5kdc[14582](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008454, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20260806 09:27:35.127131 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61: Bootstrap starting.
I20260806 09:27:35.132733 14702 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:35.134524 14702 log.cc:824] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:35.136875 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:35.109006 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:37341 (local address 127.0.0.1:41381)
0806 09:27:35.109998 (+ 992us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:35.110015 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:35.110073 (+ 58us) server_negotiation.cc:407] Connection header received
0806 09:27:35.112110 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:35.112139 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:35.112644 (+ 505us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:35.113028 (+ 384us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:35.115410 (+ 2382us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:35.116938 (+ 1528us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:35.119072 (+ 2134us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:35.119898 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:35.120065 (+ 167us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:35.126330 (+ 6265us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:35.126340 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:35.126361 (+ 21us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:35.126414 (+ 53us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:35.130420 (+ 4006us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:35.132193 (+ 1773us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:35.132206 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:35.132221 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:35.132442 (+ 221us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:35.133206 (+ 764us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:35.133210 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:35.133212 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:35.133914 (+ 702us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:35.134207 (+ 293us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:35.135136 (+ 929us) server_negotiation.cc:299] Negotiation successful
0806 09:27:35.135481 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":717,"thread_start_us":421,"threads_started":1}
I20260806 09:27:35.139324 14702 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61: No bootstrap required, opened a new log
I20260806 09:27:35.141522 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:35.150637 14702 raft_consensus.cc:348] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9e5bd27018422784a0017370ca3f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41381 } }
I20260806 09:27:35.151355 14702 raft_consensus.cc:374] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:35.151499 14702 raft_consensus.cc:729] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ec9e5bd27018422784a0017370ca3f61, State: Initialized, Role: FOLLOWER
I20260806 09:27:35.152376 14702 consensus_queue.cc:260] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9e5bd27018422784a0017370ca3f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41381 } }
I20260806 09:27:35.152782 14702 raft_consensus.cc:388] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:35.152987 14702 raft_consensus.cc:482] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:35.153254 14702 raft_consensus.cc:3037] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:35.156518 14702 raft_consensus.cc:504] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9e5bd27018422784a0017370ca3f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41381 } }
I20260806 09:27:35.157361 14702 leader_election.cc:302] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ec9e5bd27018422784a0017370ca3f61; no voters:
I20260806 09:27:35.158362 14702 leader_election.cc:288] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:35.158704 14716 raft_consensus.cc:2781] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:35.160495 14716 raft_consensus.cc:686] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [term 1 LEADER]: Becoming Leader. State: Replica: ec9e5bd27018422784a0017370ca3f61, State: Running, Role: LEADER
I20260806 09:27:35.161291 14716 consensus_queue.cc:237] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9e5bd27018422784a0017370ca3f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41381 } }
I20260806 09:27:35.161803 14702 sys_catalog.cc:564] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:35.164789 14717 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ec9e5bd27018422784a0017370ca3f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9e5bd27018422784a0017370ca3f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41381 } } }
I20260806 09:27:35.165249 14717 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:35.166973 14705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:35.146926 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:49401 (local address 127.0.0.1:41381)
0806 09:27:35.147263 (+ 337us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:35.147271 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:35.147398 (+ 127us) server_negotiation.cc:407] Connection header received
0806 09:27:35.147566 (+ 168us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:35.147573 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:35.147689 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:35.147882 (+ 193us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:35.149387 (+ 1505us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:35.150663 (+ 1276us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:35.152343 (+ 1680us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:35.153201 (+ 858us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:35.153400 (+ 199us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:35.156267 (+ 2867us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:35.156276 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:35.156282 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:35.156349 (+ 67us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:35.161024 (+ 4675us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:35.162412 (+ 1388us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:35.162420 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:35.162427 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:35.162671 (+ 244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:35.163500 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:35.163509 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:35.163513 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:35.164071 (+ 558us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:35.164486 (+ 415us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:35.166042 (+ 1556us) server_negotiation.cc:299] Negotiation successful
0806 09:27:35.166366 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":162}
I20260806 09:27:35.170086 14718 sys_catalog.cc:455] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ec9e5bd27018422784a0017370ca3f61. Latest consensus state: current_term: 1 leader_uuid: "ec9e5bd27018422784a0017370ca3f61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ec9e5bd27018422784a0017370ca3f61" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 41381 } } }
I20260806 09:27:35.171778 14718 sys_catalog.cc:458] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:35.173626 14721 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:35.180833 14721 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:35.192438 14721 catalog_manager.cc:1347] Generated new cluster ID: e7d6f9bb04b0401da183612735760010
I20260806 09:27:35.192643 14721 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:35.209355 14721 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:35.211208 14721 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:35.219826 14721 catalog_manager.cc:5868] T 00000000000000000000000000000000 P ec9e5bd27018422784a0017370ca3f61: Generated new TSK 0
I20260806 09:27:35.221097 14721 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:35.233695 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 14602
I20260806 09:27:35.235359 14696 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:35.341138 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 14602
W20260806 09:27:35.370510 14580 connection.cc:535] client connection to 127.0.0.1:41381 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:41381): Connection reset by peer (error 104)
2026-08-06T09:27:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2570 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:35 dist-test-slave-h3ft krb5kdc[14743](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:35 dist-test-slave-h3ft krb5kdc[14743](info): set up 2 sockets
Aug 06 09:27:35 dist-test-slave-h3ft krb5kdc[14743](info): commencing operation
krb5kdc: starting...
W20260806 09:27:36.822402 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.424s user 0.000s sys 0.070s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:37 dist-test-slave-h3ft krb5kdc[14743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008457, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:37Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:27:37.235379 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:38359
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:46593
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:37.680397 14762 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:37.680812 14762 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:37.680917 14762 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:37.688988 14762 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:37.689148 14762 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:37.689200 14762 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:37.689272 14762 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:37.689323 14762 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:37.700613 14762 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38359
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:46593
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:46593
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:37.702360 14762 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:37.703984 14762 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:37.719305 14768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:37.719339 14771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:37.719992 14769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:37.721000 14762 server_base.cc:1034] running on GCE node
I20260806 09:27:37.722393 14762 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:37.724572 14762 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:37.725839 14762 hybrid_clock.cc:648] HybridClock initialized: now 1786008457725790 us; error 40 us; skew 500 ppm
I20260806 09:27:37.726315 14762 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:37 dist-test-slave-h3ft krb5kdc[14743](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008457, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:37.730444 14762 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:27:37.732694 14762 webserver.cc:460] Webserver started at http://127.0.0.1:35833/ using document root <none> and password file <none>
I20260806 09:27:37.733646 14762 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:37.733834 14762 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:37.734246 14762 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:37.738718 14762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "bb01715d0600463492934cb7da52e952"
format_stamp: "Formatted at 2026-08-06 09:27:37 on dist-test-slave-h3ft"
server_key: "89643a0bbd546dae0ac8d759464537ae"
server_key_iv: "bf22d35130f829dda4c655e88bd95cef"
server_key_version: "encryptionkey@0"
I20260806 09:27:37.739913 14762 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "bb01715d0600463492934cb7da52e952"
format_stamp: "Formatted at 2026-08-06 09:27:37 on dist-test-slave-h3ft"
server_key: "89643a0bbd546dae0ac8d759464537ae"
server_key_iv: "bf22d35130f829dda4c655e88bd95cef"
server_key_version: "encryptionkey@0"
I20260806 09:27:37.747551 14762 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.002s sys 0.005s
I20260806 09:27:37.752403 14779 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:37.754352 14762 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:27:37.754606 14762 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "bb01715d0600463492934cb7da52e952"
format_stamp: "Formatted at 2026-08-06 09:27:37 on dist-test-slave-h3ft"
server_key: "89643a0bbd546dae0ac8d759464537ae"
server_key_iv: "bf22d35130f829dda4c655e88bd95cef"
server_key_version: "encryptionkey@0"
I20260806 09:27:37.754842 14762 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:37.781713 14762 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:37.786978 14762 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:37.787417 14762 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:37.810947 14762 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:46593
I20260806 09:27:37.811146 14853 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:46593 every 8 connection(s)
I20260806 09:27:37.812599 14762 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:37.817731 14856 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:37.818256 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 14762
I20260806 09:27:37.818472 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:37.818971 16730 external_mini_cluster.cc:1428] Setting key a34e1021977e478420e2fd736c6f1d84
I20260806 09:27:37.834498 14856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952: Bootstrap starting.
I20260806 09:27:37.835906 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:37.821332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:37293 (local address 127.0.0.1:46593)
0806 09:27:37.822279 (+ 947us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:37.822292 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:37.822352 (+ 60us) server_negotiation.cc:407] Connection header received
0806 09:27:37.824944 (+ 2592us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:37.824974 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:37.825470 (+ 496us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:37.825832 (+ 362us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:37.826666 (+ 834us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:37.828097 (+ 1431us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:37.830236 (+ 2139us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:37.831070 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:37.831262 (+ 192us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:37.832910 (+ 1648us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:37.832919 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:37.832936 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:37.833623 (+ 687us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:37.833724 (+ 101us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:37.833752 (+ 28us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:37.833899 (+ 147us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:37.834309 (+ 410us) server_negotiation.cc:299] Negotiation successful
0806 09:27:37.834550 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":688,"thread_start_us":446,"threads_started":1}
I20260806 09:27:37.839190 14856 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:37.840610 14856 log.cc:824] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:37.841292 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:37.844370 14856 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952: No bootstrap required, opened a new log
I20260806 09:27:37.854956 14860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:37.846739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:56423 (local address 127.0.0.1:46593)
0806 09:27:37.847064 (+ 325us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:37.847072 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:37.847591 (+ 519us) server_negotiation.cc:407] Connection header received
0806 09:27:37.847945 (+ 354us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:37.847952 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:37.848065 (+ 113us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:37.848223 (+ 158us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:37.848830 (+ 607us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:37.850091 (+ 1261us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:37.851676 (+ 1585us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:37.852457 (+ 781us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:37.852617 (+ 160us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:37.854015 (+ 1398us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:37.854023 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:37.854028 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:37.854056 (+ 28us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:37.854136 (+ 80us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:37.854160 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:37.854295 (+ 135us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:37.854519 (+ 224us) server_negotiation.cc:299] Negotiation successful
0806 09:27:37.854588 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20260806 09:27:37.855655 14856 raft_consensus.cc:348] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb01715d0600463492934cb7da52e952" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260806 09:27:37.856254 14856 raft_consensus.cc:374] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:37.856390 14856 raft_consensus.cc:729] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bb01715d0600463492934cb7da52e952, State: Initialized, Role: FOLLOWER
I20260806 09:27:37.857187 14856 consensus_queue.cc:260] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb01715d0600463492934cb7da52e952" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260806 09:27:37.857445 14856 raft_consensus.cc:388] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:37.857569 14856 raft_consensus.cc:482] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:37.857712 14856 raft_consensus.cc:3037] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:37.860410 14856 raft_consensus.cc:504] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb01715d0600463492934cb7da52e952" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260806 09:27:37.861224 14856 leader_election.cc:302] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bb01715d0600463492934cb7da52e952; no voters:
I20260806 09:27:37.862123 14856 leader_election.cc:288] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:37.862321 14870 raft_consensus.cc:2781] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:37.865031 14856 sys_catalog.cc:564] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:37.865499 14870 raft_consensus.cc:686] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [term 1 LEADER]: Becoming Leader. State: Replica: bb01715d0600463492934cb7da52e952, State: Running, Role: LEADER
I20260806 09:27:37.866180 14870 consensus_queue.cc:237] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb01715d0600463492934cb7da52e952" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } }
I20260806 09:27:37.869589 14872 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bb01715d0600463492934cb7da52e952. Latest consensus state: current_term: 1 leader_uuid: "bb01715d0600463492934cb7da52e952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb01715d0600463492934cb7da52e952" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260806 09:27:37.869797 14871 sys_catalog.cc:455] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bb01715d0600463492934cb7da52e952" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bb01715d0600463492934cb7da52e952" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 46593 } } }
I20260806 09:27:37.870277 14871 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:37.870291 14872 sys_catalog.cc:458] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:37.871862 14883 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:37.879444 14883 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:37.891968 14883 catalog_manager.cc:1347] Generated new cluster ID: c413a69d498b43cda66f4d3cf2edf0d4
I20260806 09:27:37.892135 14883 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:37.913811 14883 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:37.915349 14883 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:37.922972 14883 catalog_manager.cc:5868] T 00000000000000000000000000000000 P bb01715d0600463492934cb7da52e952: Generated new TSK 0
I20260806 09:27:37.923885 14883 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:37.930138 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 14762
W20260806 09:27:37.931589 14849 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.166:37293
W20260806 09:27:37.932435 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 14762: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:27:37.932611 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 14762
W20260806 09:27:37.958901 14741 connection.cc:535] client connection to 127.0.0.1:46593 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:46593): Connection reset by peer (error 104)
2026-08-06T09:27:37Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2588 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:38 dist-test-slave-h3ft krb5kdc[14894](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:38 dist-test-slave-h3ft krb5kdc[14894](info): set up 2 sockets
Aug 06 09:27:38 dist-test-slave-h3ft krb5kdc[14894](info): commencing operation
krb5kdc: starting...
W20260806 09:27:39.316830 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.337s user 0.003s sys 0.062s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:39 dist-test-slave-h3ft krb5kdc[14894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008459, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:39Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:39Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab.
I20260806 09:27:39.736274 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43699
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.0.0.1:46035
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43699
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:40.122704 14912 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:40.123278 14912 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:40.123430 14912 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:40.130808 14912 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:40.130936 14912 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:40.130985 14912 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:40.131162 14912 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:40.131234 14912 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:40.141453 14912 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:46035
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43699
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/krb5kdc/kudu_127.0.0.1.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43699
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:40.142961 14912 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:40.144423 14912 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:40.158190 14918 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:40.158063 14926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:40.158420 14919 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:40.160450 14912 server_base.cc:1034] running on GCE node
I20260806 09:27:40.161377 14912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:40.162966 14912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:40.164294 14912 hybrid_clock.cc:648] HybridClock initialized: now 1786008460164239 us; error 39 us; skew 500 ppm
I20260806 09:27:40.164803 14912 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:40 dist-test-slave-h3ft krb5kdc[14894](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008460, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:40.172359 14912 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20260806 09:27:40.174281 14912 webserver.cc:460] Webserver started at http://127.0.0.1:44343/ using document root <none> and password file <none>
I20260806 09:27:40.175133 14912 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:40.175294 14912 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:40.175635 14912 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:40.178807 14912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "88fe7351f1eb4ad3b846c5ce4e90607e"
format_stamp: "Formatted at 2026-08-06 09:27:40 on dist-test-slave-h3ft"
server_key: "e95834d2d71b1667f3da4c6206f56a35"
server_key_iv: "772f47dac096ef01b28d2cdfc42dd8f1"
server_key_version: "encryptionkey@0"
I20260806 09:27:40.179648 14912 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "88fe7351f1eb4ad3b846c5ce4e90607e"
format_stamp: "Formatted at 2026-08-06 09:27:40 on dist-test-slave-h3ft"
server_key: "e95834d2d71b1667f3da4c6206f56a35"
server_key_iv: "772f47dac096ef01b28d2cdfc42dd8f1"
server_key_version: "encryptionkey@0"
I20260806 09:27:40.185456 14912 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:27:40.189460 14934 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:40.191183 14912 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:27:40.191411 14912 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "88fe7351f1eb4ad3b846c5ce4e90607e"
format_stamp: "Formatted at 2026-08-06 09:27:40 on dist-test-slave-h3ft"
server_key: "e95834d2d71b1667f3da4c6206f56a35"
server_key_iv: "772f47dac096ef01b28d2cdfc42dd8f1"
server_key_version: "encryptionkey@0"
I20260806 09:27:40.191625 14912 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:40.197849 14912 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:40.202491 14912 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:40.202822 14912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:40.225471 14912 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43699
I20260806 09:27:40.225590 15000 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.0.0.1:43699 every 8 connection(s)
I20260806 09:27:40.227028 14912 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:40.229884 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 14912
I20260806 09:27:40.230036 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:40.230561 16730 external_mini_cluster.cc:1428] Setting key c3721ef8fd313c4dd9f066482cdf401f
I20260806 09:27:40.232401 15005 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:40.239743 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:40.232786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:60395 (local address 127.0.0.1:43699)
0806 09:27:40.233669 (+ 883us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:40.233682 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:40.233727 (+ 45us) server_negotiation.cc:407] Connection header received
0806 09:27:40.235526 (+ 1799us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:40.235549 (+ 23us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:40.235918 (+ 369us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:40.236294 (+ 376us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:40.237691 (+ 1397us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:40.237698 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:40.237717 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:40.237736 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:40.237831 (+ 95us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:40.237855 (+ 24us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:40.238078 (+ 223us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:40.238254 (+ 176us) server_negotiation.cc:299] Negotiation successful
0806 09:27:40.238516 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":639,"thread_start_us":373,"threads_started":1}
I20260806 09:27:40.244577 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:40.247416 15005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e: Bootstrap starting.
I20260806 09:27:40.251430 15005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:40.252720 15005 log.cc:824] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:40.253037 15009 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:40.249675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.166:60161 (local address 127.0.0.1:43699)
0806 09:27:40.249954 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:40.249965 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:40.250609 (+ 644us) server_negotiation.cc:407] Connection header received
0806 09:27:40.250813 (+ 204us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:40.250826 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:40.250881 (+ 55us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:40.250998 (+ 117us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:40.252147 (+ 1149us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:40.252154 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:40.252159 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:40.252179 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:40.252232 (+ 53us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:40.252249 (+ 17us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:40.252395 (+ 146us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:40.252534 (+ 139us) server_negotiation.cc:299] Negotiation successful
0806 09:27:40.252625 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20260806 09:27:40.256417 15005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e: No bootstrap required, opened a new log
I20260806 09:27:40.265151 15005 raft_consensus.cc:348] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88fe7351f1eb4ad3b846c5ce4e90607e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43699 } }
I20260806 09:27:40.265666 15005 raft_consensus.cc:374] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:40.265818 15005 raft_consensus.cc:729] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 88fe7351f1eb4ad3b846c5ce4e90607e, State: Initialized, Role: FOLLOWER
I20260806 09:27:40.266512 15005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88fe7351f1eb4ad3b846c5ce4e90607e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43699 } }
I20260806 09:27:40.266776 15005 raft_consensus.cc:388] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:40.266953 15005 raft_consensus.cc:482] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:40.267208 15005 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:40.269717 15005 raft_consensus.cc:504] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88fe7351f1eb4ad3b846c5ce4e90607e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43699 } }
I20260806 09:27:40.270295 15005 leader_election.cc:302] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 88fe7351f1eb4ad3b846c5ce4e90607e; no voters:
I20260806 09:27:40.271121 15005 leader_election.cc:288] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:40.271323 15018 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:40.272296 15018 raft_consensus.cc:686] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [term 1 LEADER]: Becoming Leader. State: Replica: 88fe7351f1eb4ad3b846c5ce4e90607e, State: Running, Role: LEADER
I20260806 09:27:40.273021 15018 consensus_queue.cc:237] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88fe7351f1eb4ad3b846c5ce4e90607e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43699 } }
I20260806 09:27:40.273705 15005 sys_catalog.cc:564] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:40.277455 15019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "88fe7351f1eb4ad3b846c5ce4e90607e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88fe7351f1eb4ad3b846c5ce4e90607e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43699 } } }
I20260806 09:27:40.277719 15022 sys_catalog.cc:455] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 88fe7351f1eb4ad3b846c5ce4e90607e. Latest consensus state: current_term: 1 leader_uuid: "88fe7351f1eb4ad3b846c5ce4e90607e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "88fe7351f1eb4ad3b846c5ce4e90607e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43699 } } }
I20260806 09:27:40.278079 15019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:40.278126 15022 sys_catalog.cc:458] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:40.280772 15031 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:40.286599 15031 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:40.297434 15031 catalog_manager.cc:1347] Generated new cluster ID: a8e6a8840e62411abd9d7ed5e0a48260
I20260806 09:27:40.297580 15031 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:40.309530 15031 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:40.310205 15031 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:40.326436 15031 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 88fe7351f1eb4ad3b846c5ce4e90607e: Generated new TSK 0
I20260806 09:27:40.327567 15031 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:40.346056 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 14912
W20260806 09:27:40.347448 14995 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.166:60395
W20260806 09:27:40.348373 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 14912: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:27:40.348516 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 14912
2026-08-06T09:27:40Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2418 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:40 dist-test-slave-h3ft krb5kdc[15050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:40 dist-test-slave-h3ft krb5kdc[15050](info): set up 2 sockets
Aug 06 09:27:40 dist-test-slave-h3ft krb5kdc[15050](info): commencing operation
krb5kdc: starting...
W20260806 09:27:41.705084 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.307s user 0.001s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:41 dist-test-slave-h3ft krb5kdc[15050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:42Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:27:42.124171 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:37887
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:46705
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:37887
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:42.549398 15069 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:42.549808 15069 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:42.549923 15069 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:42.558058 15069 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:42.558213 15069 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:42.558267 15069 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:42.558331 15069 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:42.558372 15069 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:42.569976 15069 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:46705
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:37887
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:37887
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:42.571777 15069 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:42.573398 15069 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:42.587687 15079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:42.587764 15076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:42.587826 15074 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:42.588840 15069 server_base.cc:1034] running on GCE node
I20260806 09:27:42.589862 15069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:42.591955 15069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:42.593218 15069 hybrid_clock.cc:648] HybridClock initialized: now 1786008462593162 us; error 47 us; skew 500 ppm
I20260806 09:27:42.593628 15069 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:42 dist-test-slave-h3ft krb5kdc[15050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008462, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:42.597579 15069 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:27:42.599701 15069 webserver.cc:460] Webserver started at http://127.16.86.190:46733/ using document root <none> and password file <none>
I20260806 09:27:42.600591 15069 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:42.600768 15069 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:42.601171 15069 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:42.605051 15069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "5bd975ec97ae4f01904bf20325ea3e4e"
format_stamp: "Formatted at 2026-08-06 09:27:42 on dist-test-slave-h3ft"
server_key: "a97b603942261959b687d45e046c1853"
server_key_iv: "47b5bbeaeabe07b326cc38621f14e067"
server_key_version: "encryptionkey@0"
I20260806 09:27:42.606006 15069 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "5bd975ec97ae4f01904bf20325ea3e4e"
format_stamp: "Formatted at 2026-08-06 09:27:42 on dist-test-slave-h3ft"
server_key: "a97b603942261959b687d45e046c1853"
server_key_iv: "47b5bbeaeabe07b326cc38621f14e067"
server_key_version: "encryptionkey@0"
I20260806 09:27:42.612877 15069 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.004s
I20260806 09:27:42.617419 15091 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:42.619521 15069 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260806 09:27:42.619822 15069 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "5bd975ec97ae4f01904bf20325ea3e4e"
format_stamp: "Formatted at 2026-08-06 09:27:42 on dist-test-slave-h3ft"
server_key: "a97b603942261959b687d45e046c1853"
server_key_iv: "47b5bbeaeabe07b326cc38621f14e067"
server_key_version: "encryptionkey@0"
I20260806 09:27:42.620126 15069 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:42.648218 15069 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:42.654206 15069 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:42.654654 15069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:42.681269 15069 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:37887
I20260806 09:27:42.681322 15163 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:37887 every 8 connection(s)
I20260806 09:27:42.683295 15069 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:42.687072 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 15069
I20260806 09:27:42.687317 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:42.687866 16730 external_mini_cluster.cc:1428] Setting key 83514a13680c33739cadfe742e463279
I20260806 09:27:42.688037 15164 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:42.704731 15164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e: Bootstrap starting.
Aug 06 09:27:42 dist-test-slave-h3ft krb5kdc[15050](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008461, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:27:42.708959 15164 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:42.710330 15164 log.cc:824] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:42.714630 15164 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e: No bootstrap required, opened a new log
I20260806 09:27:42.718092 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:42.690371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41815 (local address 127.16.86.190:37887)
0806 09:27:42.691524 (+ 1153us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:42.691542 (+ 18us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:42.691599 (+ 57us) server_negotiation.cc:407] Connection header received
0806 09:27:42.693635 (+ 2036us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:42.693663 (+ 28us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:42.694190 (+ 527us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:42.694623 (+ 433us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:42.696712 (+ 2089us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:42.698276 (+ 1564us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:42.700247 (+ 1971us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:42.701157 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:42.701327 (+ 170us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:42.707233 (+ 5906us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:42.707242 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:42.707262 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:42.707326 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:42.711894 (+ 4568us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:42.713262 (+ 1368us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:42.713271 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:42.713286 (+ 15us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:42.713773 (+ 487us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:42.714444 (+ 671us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:42.714448 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:42.714450 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:42.715340 (+ 890us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:42.715668 (+ 328us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:42.716232 (+ 564us) server_negotiation.cc:299] Negotiation successful
0806 09:27:42.716716 (+ 484us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":848,"thread_start_us":559,"threads_started":1}
I20260806 09:27:42.723412 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:42.724375 15164 raft_consensus.cc:348] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd975ec97ae4f01904bf20325ea3e4e" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37887 } }
I20260806 09:27:42.724778 15164 raft_consensus.cc:374] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:42.725008 15164 raft_consensus.cc:729] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5bd975ec97ae4f01904bf20325ea3e4e, State: Initialized, Role: FOLLOWER
I20260806 09:27:42.725672 15164 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd975ec97ae4f01904bf20325ea3e4e" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37887 } }
I20260806 09:27:42.725893 15164 raft_consensus.cc:388] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:42.726079 15164 raft_consensus.cc:482] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:42.726267 15164 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:42.728829 15164 raft_consensus.cc:504] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd975ec97ae4f01904bf20325ea3e4e" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37887 } }
I20260806 09:27:42.729482 15164 leader_election.cc:302] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5bd975ec97ae4f01904bf20325ea3e4e; no voters:
I20260806 09:27:42.731185 15164 leader_election.cc:288] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:42.731369 15180 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:42.733439 15164 sys_catalog.cc:564] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:42.735131 15180 raft_consensus.cc:686] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [term 1 LEADER]: Becoming Leader. State: Replica: 5bd975ec97ae4f01904bf20325ea3e4e, State: Running, Role: LEADER
I20260806 09:27:42.736327 15180 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd975ec97ae4f01904bf20325ea3e4e" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37887 } }
I20260806 09:27:42.739566 15181 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5bd975ec97ae4f01904bf20325ea3e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd975ec97ae4f01904bf20325ea3e4e" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37887 } } }
I20260806 09:27:42.739986 15181 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:42.741887 15193 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:42.742129 15182 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5bd975ec97ae4f01904bf20325ea3e4e. Latest consensus state: current_term: 1 leader_uuid: "5bd975ec97ae4f01904bf20325ea3e4e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5bd975ec97ae4f01904bf20325ea3e4e" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 37887 } } }
I20260806 09:27:42.742412 15182 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:42.748772 15193 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:42.756743 15168 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:42.729913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52497 (local address 127.16.86.190:37887)
0806 09:27:42.730149 (+ 236us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:42.730154 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:42.730767 (+ 613us) server_negotiation.cc:407] Connection header received
0806 09:27:42.732363 (+ 1596us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:42.732370 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:42.732468 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:42.732580 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:42.734762 (+ 2182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:42.737184 (+ 2422us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:42.740014 (+ 2830us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:42.741190 (+ 1176us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:42.741353 (+ 163us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:42.746313 (+ 4960us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:42.746349 (+ 36us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:42.746357 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:42.746428 (+ 71us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:42.750672 (+ 4244us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:42.752321 (+ 1649us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:42.752332 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:42.752340 (+ 8us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:42.752614 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:42.753526 (+ 912us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:42.753537 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:42.753543 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:42.754346 (+ 803us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:42.754807 (+ 461us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:42.755916 (+ 1109us) server_negotiation.cc:299] Negotiation successful
0806 09:27:42.756251 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20260806 09:27:42.761685 15193 catalog_manager.cc:1347] Generated new cluster ID: f86b123ad1b144eeb997e400524d3e79
I20260806 09:27:42.761829 15193 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:42.789119 15193 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:42.791093 15193 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:42.803339 15193 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 5bd975ec97ae4f01904bf20325ea3e4e: Generated new TSK 0
I20260806 09:27:42.804615 15193 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:42.814769 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 15069
I20260806 09:27:42.816278 15158 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:42.928617 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 15069
W20260806 09:27:42.962126 15047 connection.cc:535] client connection to 127.16.86.190:37887 recv error: Network error: failed to read from TLS socket (remote: 127.16.86.190:37887): Connection reset by peer (error 104)
2026-08-06T09:27:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2584 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:43 dist-test-slave-h3ft krb5kdc[15209](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:43 dist-test-slave-h3ft krb5kdc[15209](info): set up 2 sockets
Aug 06 09:27:43 dist-test-slave-h3ft krb5kdc[15209](info): commencing operation
krb5kdc: starting...
W20260806 09:27:44.303153 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.321s user 0.000s sys 0.068s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:44 dist-test-slave-h3ft krb5kdc[15209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008464, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:44Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:27:44.715948 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:36657
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:40755
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:36657
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:45.129313 15223 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:45.129642 15223 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:45.129827 15223 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:45.137265 15223 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:45.137409 15223 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:45.137506 15223 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:45.137637 15223 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:45.137708 15223 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:45.148108 15223 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:40755
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:36657
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:36657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:45.149598 15223 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:45.151144 15223 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:45.164978 15231 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:45.165117 15232 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:45.166090 15234 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:45.166613 15223 server_base.cc:1034] running on GCE node
I20260806 09:27:45.167806 15223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:45.169799 15223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:45.171087 15223 hybrid_clock.cc:648] HybridClock initialized: now 1786008465170956 us; error 78 us; skew 500 ppm
I20260806 09:27:45.171489 15223 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:45 dist-test-slave-h3ft krb5kdc[15209](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008465, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:45.181439 15223 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:27:45.183660 15223 webserver.cc:460] Webserver started at http://127.16.86.190:34083/ using document root <none> and password file <none>
I20260806 09:27:45.184610 15223 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:45.184770 15223 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:45.185175 15223 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:45.189160 15223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "782ae0d2f28b4d2f8e14984a4c7c6f8f"
format_stamp: "Formatted at 2026-08-06 09:27:45 on dist-test-slave-h3ft"
server_key: "fdd5d550ceba2bde51f2ccf7ca9859c4"
server_key_iv: "2325436c5df597fdbec502260b70ad34"
server_key_version: "encryptionkey@0"
I20260806 09:27:45.190313 15223 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "782ae0d2f28b4d2f8e14984a4c7c6f8f"
format_stamp: "Formatted at 2026-08-06 09:27:45 on dist-test-slave-h3ft"
server_key: "fdd5d550ceba2bde51f2ccf7ca9859c4"
server_key_iv: "2325436c5df597fdbec502260b70ad34"
server_key_version: "encryptionkey@0"
I20260806 09:27:45.197602 15223 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260806 09:27:45.202642 15243 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:45.204749 15223 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:27:45.205049 15223 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "782ae0d2f28b4d2f8e14984a4c7c6f8f"
format_stamp: "Formatted at 2026-08-06 09:27:45 on dist-test-slave-h3ft"
server_key: "fdd5d550ceba2bde51f2ccf7ca9859c4"
server_key_iv: "2325436c5df597fdbec502260b70ad34"
server_key_version: "encryptionkey@0"
I20260806 09:27:45.205328 15223 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:45.222079 15223 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:45.227341 15223 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:45.227702 15223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:45.251682 15223 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:36657
I20260806 09:27:45.251771 15320 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:36657 every 8 connection(s)
I20260806 09:27:45.253716 15223 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:45.258445 15321 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:45.258868 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 15223
I20260806 09:27:45.259176 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:45.259796 16730 external_mini_cluster.cc:1428] Setting key d7ffff7ae49001f47bd8e6dde0b273ee
I20260806 09:27:45.274403 15321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f: Bootstrap starting.
I20260806 09:27:45.277429 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:45.262391 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46021 (local address 127.16.86.190:36657)
0806 09:27:45.263575 (+ 1184us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:45.263591 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:45.263654 (+ 63us) server_negotiation.cc:407] Connection header received
0806 09:27:45.265821 (+ 2167us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:45.265858 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:45.266453 (+ 595us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:45.266931 (+ 478us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:45.267719 (+ 788us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:45.269209 (+ 1490us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:45.271369 (+ 2160us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:45.272332 (+ 963us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:45.272513 (+ 181us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:45.274316 (+ 1803us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:45.274325 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:45.274344 (+ 19us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:45.275226 (+ 882us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:45.275332 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:45.275363 (+ 31us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:45.275530 (+ 167us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:45.275847 (+ 317us) server_negotiation.cc:299] Negotiation successful
0806 09:27:45.276170 (+ 323us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":885,"thread_start_us":566,"threads_started":1}
I20260806 09:27:45.278167 15321 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:45.279412 15321 log.cc:824] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:45.282629 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:45.283608 15321 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f: No bootstrap required, opened a new log
I20260806 09:27:45.294519 15321 raft_consensus.cc:348] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782ae0d2f28b4d2f8e14984a4c7c6f8f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36657 } }
I20260806 09:27:45.295305 15321 raft_consensus.cc:374] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:45.295461 15321 raft_consensus.cc:729] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 782ae0d2f28b4d2f8e14984a4c7c6f8f, State: Initialized, Role: FOLLOWER
I20260806 09:27:45.296200 15324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:45.287725 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52547 (local address 127.16.86.190:36657)
0806 09:27:45.287962 (+ 237us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:45.287969 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:45.288668 (+ 699us) server_negotiation.cc:407] Connection header received
0806 09:27:45.288918 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:45.288926 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:45.289008 (+ 82us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:45.289115 (+ 107us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:45.289820 (+ 705us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:45.290673 (+ 853us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:45.292950 (+ 2277us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:45.293447 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:45.293551 (+ 104us) server_negotiation.cc:662] Negotiated TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:45.295092 (+ 1541us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:45.295097 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:45.295100 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:45.295119 (+ 19us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:45.295219 (+ 100us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:45.295244 (+ 25us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:45.295401 (+ 157us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:45.295752 (+ 351us) server_negotiation.cc:299] Negotiation successful
0806 09:27:45.295846 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 09:27:45.296365 15321 consensus_queue.cc:260] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782ae0d2f28b4d2f8e14984a4c7c6f8f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36657 } }
I20260806 09:27:45.296732 15321 raft_consensus.cc:388] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:45.296914 15321 raft_consensus.cc:482] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:45.297205 15321 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:45.300199 15321 raft_consensus.cc:504] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782ae0d2f28b4d2f8e14984a4c7c6f8f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36657 } }
I20260806 09:27:45.300841 15321 leader_election.cc:302] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 782ae0d2f28b4d2f8e14984a4c7c6f8f; no voters:
I20260806 09:27:45.301718 15321 leader_election.cc:288] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:45.301847 15332 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:45.302897 15332 raft_consensus.cc:686] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [term 1 LEADER]: Becoming Leader. State: Replica: 782ae0d2f28b4d2f8e14984a4c7c6f8f, State: Running, Role: LEADER
I20260806 09:27:45.303725 15332 consensus_queue.cc:237] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782ae0d2f28b4d2f8e14984a4c7c6f8f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36657 } }
I20260806 09:27:45.304808 15321 sys_catalog.cc:564] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:45.307911 15334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "782ae0d2f28b4d2f8e14984a4c7c6f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782ae0d2f28b4d2f8e14984a4c7c6f8f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36657 } } }
I20260806 09:27:45.309206 15334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:45.308440 15335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 782ae0d2f28b4d2f8e14984a4c7c6f8f. Latest consensus state: current_term: 1 leader_uuid: "782ae0d2f28b4d2f8e14984a4c7c6f8f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "782ae0d2f28b4d2f8e14984a4c7c6f8f" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36657 } } }
I20260806 09:27:45.310621 15335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:45.312615 15339 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:45.317734 15339 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:45.327883 15339 catalog_manager.cc:1347] Generated new cluster ID: 304284d33b7b44ed9f14f5fda81a287f
I20260806 09:27:45.328032 15339 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:45.345921 15339 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:45.347498 15339 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:45.356200 15339 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 782ae0d2f28b4d2f8e14984a4c7c6f8f: Generated new TSK 0
I20260806 09:27:45.357215 15339 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:45.375173 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 15223
W20260806 09:27:45.376853 15311 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:46021
W20260806 09:27:45.377935 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 15223: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:27:45.378141 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 15223
W20260806 09:27:45.409403 15205 connection.cc:535] client connection to 127.16.86.190:36657 recv error: Network error: failed to read from TLS socket (remote: 127.16.86.190:36657): Connection reset by peer (error 104)
2026-08-06T09:27:45Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2446 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:45 dist-test-slave-h3ft krb5kdc[15358](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:45 dist-test-slave-h3ft krb5kdc[15358](info): set up 2 sockets
Aug 06 09:27:45 dist-test-slave-h3ft krb5kdc[15358](info): commencing operation
krb5kdc: starting...
W20260806 09:27:46.744352 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.315s user 0.008s sys 0.048s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:47 dist-test-slave-h3ft krb5kdc[15358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008467, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:47Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:47Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:27:47.158836 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:36477
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:32855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:36477
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:47.553102 15375 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:47.553452 15375 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:47.553546 15375 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:47.561025 15375 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:47.561158 15375 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:47.561214 15375 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:47.561282 15375 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:47.561337 15375 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:47.571493 15375 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:32855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:36477
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:36477
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:47.572876 15375 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:47.574218 15375 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:47.587745 15381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:47.587862 15382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:47.587885 15384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:47.588979 15375 server_base.cc:1034] running on GCE node
I20260806 09:27:47.589973 15375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:47.591737 15375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:47.593019 15375 hybrid_clock.cc:648] HybridClock initialized: now 1786008467592950 us; error 61 us; skew 500 ppm
I20260806 09:27:47.593509 15375 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:47 dist-test-slave-h3ft krb5kdc[15358](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008467, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:47.597404 15375 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:27:47.599459 15375 webserver.cc:460] Webserver started at http://127.16.86.190:40625/ using document root <none> and password file <none>
I20260806 09:27:47.600232 15375 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:47.600392 15375 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:47.600744 15375 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:47.604017 15375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "2756ddc1ef6a4775aa2e29652bb2d97a"
format_stamp: "Formatted at 2026-08-06 09:27:47 on dist-test-slave-h3ft"
server_key: "9fe203809e1f242de8b9c8a4b47e03d0"
server_key_iv: "d235bbac0127af443b1f453230b9cff9"
server_key_version: "encryptionkey@0"
I20260806 09:27:47.604835 15375 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "2756ddc1ef6a4775aa2e29652bb2d97a"
format_stamp: "Formatted at 2026-08-06 09:27:47 on dist-test-slave-h3ft"
server_key: "9fe203809e1f242de8b9c8a4b47e03d0"
server_key_iv: "d235bbac0127af443b1f453230b9cff9"
server_key_version: "encryptionkey@0"
I20260806 09:27:47.611181 15375 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260806 09:27:47.615772 15393 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:47.617538 15375 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:27:47.617803 15375 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "2756ddc1ef6a4775aa2e29652bb2d97a"
format_stamp: "Formatted at 2026-08-06 09:27:47 on dist-test-slave-h3ft"
server_key: "9fe203809e1f242de8b9c8a4b47e03d0"
server_key_iv: "d235bbac0127af443b1f453230b9cff9"
server_key_version: "encryptionkey@0"
I20260806 09:27:47.618044 15375 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:47.624577 15375 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:47.629321 15375 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:47.629654 15375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:47.653225 15375 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:36477
I20260806 09:27:47.653298 15462 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:36477 every 8 connection(s)
I20260806 09:27:47.654882 15375 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:47.659994 15463 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:47.663448 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 15375
I20260806 09:27:47.663712 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:47.664391 16730 external_mini_cluster.cc:1428] Setting key b5c829aab4350e07c293e28e9e5429fa
I20260806 09:27:47.675287 15466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:47.667113 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44409 (local address 127.16.86.190:36477)
0806 09:27:47.668361 (+ 1248us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:47.668394 (+ 33us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:47.668454 (+ 60us) server_negotiation.cc:407] Connection header received
0806 09:27:47.670813 (+ 2359us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:47.670849 (+ 36us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:47.671460 (+ 611us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:47.671986 (+ 526us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:47.673323 (+ 1337us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:47.673331 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:47.673349 (+ 18us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:47.673366 (+ 17us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:47.673432 (+ 66us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:47.673453 (+ 21us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:47.673756 (+ 303us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:47.673906 (+ 150us) server_negotiation.cc:299] Negotiation successful
0806 09:27:47.674125 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":879,"thread_start_us":533,"threads_started":1}
I20260806 09:27:47.676192 15463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a: Bootstrap starting.
I20260806 09:27:47.679827 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:47.680509 15463 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:47.681782 15463 log.cc:824] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:47.685830 15463 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a: No bootstrap required, opened a new log
I20260806 09:27:47.688153 15466 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:47.684909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57399 (local address 127.16.86.190:36477)
0806 09:27:47.685179 (+ 270us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:47.685186 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:47.685734 (+ 548us) server_negotiation.cc:407] Connection header received
0806 09:27:47.685950 (+ 216us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:47.685955 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:47.686039 (+ 84us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:47.686144 (+ 105us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:47.687376 (+ 1232us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:47.687380 (+ 4us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:47.687384 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:47.687400 (+ 16us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:47.687446 (+ 46us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:47.687462 (+ 16us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:47.687572 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:47.687747 (+ 175us) server_negotiation.cc:299] Negotiation successful
0806 09:27:47.687803 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
I20260806 09:27:47.697252 15463 raft_consensus.cc:348] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2756ddc1ef6a4775aa2e29652bb2d97a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36477 } }
I20260806 09:27:47.697847 15463 raft_consensus.cc:374] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:47.697984 15463 raft_consensus.cc:729] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2756ddc1ef6a4775aa2e29652bb2d97a, State: Initialized, Role: FOLLOWER
I20260806 09:27:47.698757 15463 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2756ddc1ef6a4775aa2e29652bb2d97a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36477 } }
I20260806 09:27:47.699075 15463 raft_consensus.cc:388] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:47.699246 15463 raft_consensus.cc:482] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:47.699440 15463 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:47.702097 15463 raft_consensus.cc:504] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2756ddc1ef6a4775aa2e29652bb2d97a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36477 } }
I20260806 09:27:47.702849 15463 leader_election.cc:302] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2756ddc1ef6a4775aa2e29652bb2d97a; no voters:
I20260806 09:27:47.703747 15463 leader_election.cc:288] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:47.704002 15476 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:47.704914 15476 raft_consensus.cc:686] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [term 1 LEADER]: Becoming Leader. State: Replica: 2756ddc1ef6a4775aa2e29652bb2d97a, State: Running, Role: LEADER
I20260806 09:27:47.705683 15476 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2756ddc1ef6a4775aa2e29652bb2d97a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36477 } }
I20260806 09:27:47.706494 15463 sys_catalog.cc:564] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:47.709414 15478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2756ddc1ef6a4775aa2e29652bb2d97a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2756ddc1ef6a4775aa2e29652bb2d97a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36477 } } }
I20260806 09:27:47.709913 15478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:47.710799 15481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2756ddc1ef6a4775aa2e29652bb2d97a. Latest consensus state: current_term: 1 leader_uuid: "2756ddc1ef6a4775aa2e29652bb2d97a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2756ddc1ef6a4775aa2e29652bb2d97a" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 36477 } } }
I20260806 09:27:47.711344 15481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:47.711585 15486 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:47.720851 15486 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:47.732517 15486 catalog_manager.cc:1347] Generated new cluster ID: 92aaef68fdfb475986e92b891faeb406
I20260806 09:27:47.732692 15486 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:47.759694 15486 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:47.760422 15486 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:47.770602 15486 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 2756ddc1ef6a4775aa2e29652bb2d97a: Generated new TSK 0
I20260806 09:27:47.771839 15486 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:47.787515 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 15375
W20260806 09:27:47.789086 15458 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44409
W20260806 09:27:47.790162 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 15375: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:27:47.790339 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 15375
2026-08-06T09:27:47Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2412 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:47 dist-test-slave-h3ft krb5kdc[15505](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:47 dist-test-slave-h3ft krb5kdc[15505](info): set up 2 sockets
Aug 06 09:27:47 dist-test-slave-h3ft krb5kdc[15505](info): commencing operation
krb5kdc: starting...
W20260806 09:27:49.156867 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.314s user 0.000s sys 0.069s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:49 dist-test-slave-h3ft krb5kdc[15505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:49Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:49Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:27:49.572326 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:34419
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:33235
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:34419
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:50.006732 15522 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:50.007210 15522 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:50.007458 15522 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:50.016323 15522 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:50.016561 15522 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:50.016675 15522 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:50.016830 15522 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:50.016904 15522 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:50.030627 15522 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:33235
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:34419
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:34419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:50.032352 15522 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:50.033941 15522 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:50.048432 15531 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:50.048549 15534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:50.048447 15529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:50.049949 15522 server_base.cc:1034] running on GCE node
I20260806 09:27:50.051230 15522 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:50.053417 15522 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:50.054723 15522 hybrid_clock.cc:648] HybridClock initialized: now 1786008470054638 us; error 75 us; skew 500 ppm
I20260806 09:27:50.055377 15522 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:50 dist-test-slave-h3ft krb5kdc[15505](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008470, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:50.059532 15522 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:27:50.061525 15522 webserver.cc:460] Webserver started at http://127.16.86.190:38601/ using document root <none> and password file <none>
I20260806 09:27:50.062412 15522 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:50.062534 15522 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:50.062855 15522 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:50.066717 15522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "47851f249fdf4dae9e52b788e050d9c4"
format_stamp: "Formatted at 2026-08-06 09:27:50 on dist-test-slave-h3ft"
server_key: "09ac5b73c1a5a6a1f94415f41fc68248"
server_key_iv: "16c439477da3d7f632d7d4e7908179ed"
server_key_version: "encryptionkey@0"
I20260806 09:27:50.067755 15522 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "47851f249fdf4dae9e52b788e050d9c4"
format_stamp: "Formatted at 2026-08-06 09:27:50 on dist-test-slave-h3ft"
server_key: "09ac5b73c1a5a6a1f94415f41fc68248"
server_key_iv: "16c439477da3d7f632d7d4e7908179ed"
server_key_version: "encryptionkey@0"
I20260806 09:27:50.074956 15522 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260806 09:27:50.079993 15545 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:50.082015 15522 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260806 09:27:50.082283 15522 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "47851f249fdf4dae9e52b788e050d9c4"
format_stamp: "Formatted at 2026-08-06 09:27:50 on dist-test-slave-h3ft"
server_key: "09ac5b73c1a5a6a1f94415f41fc68248"
server_key_iv: "16c439477da3d7f632d7d4e7908179ed"
server_key_version: "encryptionkey@0"
I20260806 09:27:50.082546 15522 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:50.099089 15522 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:50.105515 15522 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:50.105922 15522 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:50.129379 15522 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:34419
I20260806 09:27:50.129422 15613 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:34419 every 8 connection(s)
I20260806 09:27:50.131342 15522 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:50.134135 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 15522
I20260806 09:27:50.134379 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:50.135063 16730 external_mini_cluster.cc:1428] Setting key 23867159eb8f8c8bd36e3fde35eca862
I20260806 09:27:50.136627 15614 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Aug 06 09:27:50 dist-test-slave-h3ft krb5kdc[15505](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008469, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:27:50.152585 15614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4: Bootstrap starting.
I20260806 09:27:50.156162 15614 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:50.157320 15614 log.cc:824] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:50.160462 15614 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4: No bootstrap required, opened a new log
I20260806 09:27:50.163339 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:50.137491 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37915 (local address 127.16.86.190:34419)
0806 09:27:50.138520 (+ 1029us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:50.138535 (+ 15us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:50.138591 (+ 56us) server_negotiation.cc:407] Connection header received
0806 09:27:50.140628 (+ 2037us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:50.140658 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:50.141212 (+ 554us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:50.141671 (+ 459us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:50.143767 (+ 2096us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:50.145426 (+ 1659us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:50.147220 (+ 1794us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:50.148031 (+ 811us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:50.148185 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:50.153898 (+ 5713us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:50.153908 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:50.153928 (+ 20us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:50.153985 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:50.157928 (+ 3943us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:50.158873 (+ 945us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:50.158881 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:50.158890 (+ 9us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:50.159159 (+ 269us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:50.159613 (+ 454us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:50.159619 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:50.159622 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:50.160325 (+ 703us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:50.160700 (+ 375us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:50.161249 (+ 549us) server_negotiation.cc:299] Negotiation successful
0806 09:27:50.161832 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":493,"threads_started":1}
I20260806 09:27:50.168954 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:50.169978 15614 raft_consensus.cc:348] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47851f249fdf4dae9e52b788e050d9c4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34419 } }
I20260806 09:27:50.170399 15614 raft_consensus.cc:374] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:50.170512 15614 raft_consensus.cc:729] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 47851f249fdf4dae9e52b788e050d9c4, State: Initialized, Role: FOLLOWER
I20260806 09:27:50.171240 15614 consensus_queue.cc:260] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47851f249fdf4dae9e52b788e050d9c4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34419 } }
I20260806 09:27:50.171486 15614 raft_consensus.cc:388] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:50.171648 15614 raft_consensus.cc:482] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:50.171844 15614 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:50.174392 15614 raft_consensus.cc:504] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47851f249fdf4dae9e52b788e050d9c4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34419 } }
I20260806 09:27:50.175037 15614 leader_election.cc:302] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 47851f249fdf4dae9e52b788e050d9c4; no voters:
I20260806 09:27:50.176695 15614 leader_election.cc:288] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:50.178743 15625 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:50.180711 15625 raft_consensus.cc:686] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [term 1 LEADER]: Becoming Leader. State: Replica: 47851f249fdf4dae9e52b788e050d9c4, State: Running, Role: LEADER
I20260806 09:27:50.181358 15614 sys_catalog.cc:564] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:50.181679 15625 consensus_queue.cc:237] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47851f249fdf4dae9e52b788e050d9c4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34419 } }
I20260806 09:27:50.186445 15627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "47851f249fdf4dae9e52b788e050d9c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47851f249fdf4dae9e52b788e050d9c4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34419 } } }
I20260806 09:27:50.186923 15627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:50.186635 15628 sys_catalog.cc:455] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 47851f249fdf4dae9e52b788e050d9c4. Latest consensus state: current_term: 1 leader_uuid: "47851f249fdf4dae9e52b788e050d9c4" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "47851f249fdf4dae9e52b788e050d9c4" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 34419 } } }
I20260806 09:27:50.187222 15628 sys_catalog.cc:458] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:50.195801 15633 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:50.201222 15619 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:50.175342 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54955 (local address 127.16.86.190:34419)
0806 09:27:50.175628 (+ 286us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:50.175638 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:50.176235 (+ 597us) server_negotiation.cc:407] Connection header received
0806 09:27:50.176421 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:50.176427 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:50.176551 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:50.176712 (+ 161us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:50.178674 (+ 1962us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:50.180116 (+ 1442us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:50.182210 (+ 2094us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:50.183261 (+ 1051us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:50.183451 (+ 190us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:50.192023 (+ 8572us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:50.192028 (+ 5us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:50.192032 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:50.192076 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:50.195582 (+ 3506us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:50.196948 (+ 1366us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:50.196958 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:50.196963 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:50.197233 (+ 270us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:50.197789 (+ 556us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:50.197794 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:50.197797 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:50.198330 (+ 533us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:50.200236 (+ 1906us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:50.200424 (+ 188us) server_negotiation.cc:299] Negotiation successful
0806 09:27:50.200701 (+ 277us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 09:27:50.203658 15633 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:50.216723 15633 catalog_manager.cc:1347] Generated new cluster ID: d29cee0bed1c4aac800abe23e3771ce1
I20260806 09:27:50.216904 15633 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:50.247579 15633 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:50.249886 15633 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:50.266219 15633 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 47851f249fdf4dae9e52b788e050d9c4: Generated new TSK 0
I20260806 09:27:50.267706 15633 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:50.279812 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 15522
I20260806 09:27:50.281229 15604 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:27:50.377394 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 15522
2026-08-06T09:27:50Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2586 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:50 dist-test-slave-h3ft krb5kdc[15656](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:50 dist-test-slave-h3ft krb5kdc[15656](info): set up 2 sockets
Aug 06 09:27:50 dist-test-slave-h3ft krb5kdc[15656](info): commencing operation
krb5kdc: starting...
W20260806 09:27:51.818054 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.389s user 0.002s sys 0.067s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:52 dist-test-slave-h3ft krb5kdc[15656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008472, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:52Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:52Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:27:52.243854 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:38023
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:39129
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:38023
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:52.627490 15675 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:52.627813 15675 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:52.627988 15675 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:52.635313 15675 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:52.635435 15675 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:52.635517 15675 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:52.635628 15675 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:52.635705 15675 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:52.646178 15675 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:39129
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:38023
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:38023
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:52.647526 15675 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:52.648838 15675 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:52.663133 15682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:52.663226 15683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:52.663282 15686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:52.664019 15675 server_base.cc:1034] running on GCE node
I20260806 09:27:52.665210 15675 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:52.667222 15675 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:52.668717 15675 hybrid_clock.cc:648] HybridClock initialized: now 1786008472668657 us; error 51 us; skew 500 ppm
I20260806 09:27:52.669210 15675 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:52 dist-test-slave-h3ft krb5kdc[15656](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008472, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:52.673182 15675 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:27:52.675060 15675 webserver.cc:460] Webserver started at http://127.16.86.190:39689/ using document root <none> and password file <none>
I20260806 09:27:52.675872 15675 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:52.676040 15675 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:52.676385 15675 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:52.679757 15675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "85a6a7279939455ba36d3de1efc06a35"
format_stamp: "Formatted at 2026-08-06 09:27:52 on dist-test-slave-h3ft"
server_key: "ed0bafbe6ea528761042f923453b7f54"
server_key_iv: "870e3c86026c75b7da755a80cedc4eb3"
server_key_version: "encryptionkey@0"
I20260806 09:27:52.680619 15675 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "85a6a7279939455ba36d3de1efc06a35"
format_stamp: "Formatted at 2026-08-06 09:27:52 on dist-test-slave-h3ft"
server_key: "ed0bafbe6ea528761042f923453b7f54"
server_key_iv: "870e3c86026c75b7da755a80cedc4eb3"
server_key_version: "encryptionkey@0"
I20260806 09:27:52.686813 15675 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260806 09:27:52.690917 15695 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:52.692724 15675 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:27:52.692955 15675 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "85a6a7279939455ba36d3de1efc06a35"
format_stamp: "Formatted at 2026-08-06 09:27:52 on dist-test-slave-h3ft"
server_key: "ed0bafbe6ea528761042f923453b7f54"
server_key_iv: "870e3c86026c75b7da755a80cedc4eb3"
server_key_version: "encryptionkey@0"
I20260806 09:27:52.693176 15675 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:52.726917 15675 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:52.731999 15675 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:52.732340 15675 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:52.755247 15675 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:38023
I20260806 09:27:52.755333 15761 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:38023 every 8 connection(s)
I20260806 09:27:52.756837 15675 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:52.758494 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 15675
I20260806 09:27:52.758702 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:52.759186 16730 external_mini_cluster.cc:1428] Setting key c7218594448f025c3a68d3096f11557e
I20260806 09:27:52.763365 15763 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:52.778021 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:52.761582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44491 (local address 127.16.86.190:38023)
0806 09:27:52.762721 (+ 1139us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:52.762743 (+ 22us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:52.762817 (+ 74us) server_negotiation.cc:407] Connection header received
0806 09:27:52.765670 (+ 2853us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:52.765707 (+ 37us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:52.766369 (+ 662us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:52.766958 (+ 589us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:52.767683 (+ 725us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:52.770079 (+ 2396us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:52.771663 (+ 1584us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:52.772916 (+ 1253us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:52.773136 (+ 220us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:52.774539 (+ 1403us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:52.774553 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:52.774577 (+ 24us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:52.775501 (+ 924us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:52.775621 (+ 120us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:52.775664 (+ 43us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:52.775839 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:52.775999 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 09:27:52.776355 (+ 356us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":447,"threads_started":1}
I20260806 09:27:52.781248 15763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35: Bootstrap starting.
I20260806 09:27:52.783437 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:52.785638 15763 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:52.787161 15763 log.cc:824] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:52.791262 15763 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35: No bootstrap required, opened a new log
I20260806 09:27:52.795889 15769 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:52.788861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55373 (local address 127.16.86.190:38023)
0806 09:27:52.789053 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:52.789058 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:52.789721 (+ 663us) server_negotiation.cc:407] Connection header received
0806 09:27:52.789975 (+ 254us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:52.789980 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:52.790048 (+ 68us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:52.790175 (+ 127us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:52.790730 (+ 555us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:52.791640 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:52.792988 (+ 1348us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:52.793526 (+ 538us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:52.793709 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:52.795052 (+ 1343us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:52.795059 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:52.795064 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:52.795098 (+ 34us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:52.795149 (+ 51us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:52.795167 (+ 18us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:52.795270 (+ 103us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:52.795430 (+ 160us) server_negotiation.cc:299] Negotiation successful
0806 09:27:52.795534 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20260806 09:27:52.801110 15763 raft_consensus.cc:348] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a6a7279939455ba36d3de1efc06a35" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38023 } }
I20260806 09:27:52.801740 15763 raft_consensus.cc:374] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:52.801880 15763 raft_consensus.cc:729] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 85a6a7279939455ba36d3de1efc06a35, State: Initialized, Role: FOLLOWER
I20260806 09:27:52.802681 15763 consensus_queue.cc:260] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a6a7279939455ba36d3de1efc06a35" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38023 } }
I20260806 09:27:52.802942 15763 raft_consensus.cc:388] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:52.803138 15763 raft_consensus.cc:482] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:52.803309 15763 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:52.805917 15763 raft_consensus.cc:504] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a6a7279939455ba36d3de1efc06a35" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38023 } }
I20260806 09:27:52.806690 15763 leader_election.cc:302] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 85a6a7279939455ba36d3de1efc06a35; no voters:
I20260806 09:27:52.807536 15763 leader_election.cc:288] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:52.807809 15778 raft_consensus.cc:2781] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:52.808923 15778 raft_consensus.cc:686] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [term 1 LEADER]: Becoming Leader. State: Replica: 85a6a7279939455ba36d3de1efc06a35, State: Running, Role: LEADER
I20260806 09:27:52.809607 15778 consensus_queue.cc:237] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a6a7279939455ba36d3de1efc06a35" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38023 } }
I20260806 09:27:52.810320 15763 sys_catalog.cc:564] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:52.813364 15780 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "85a6a7279939455ba36d3de1efc06a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a6a7279939455ba36d3de1efc06a35" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38023 } } }
I20260806 09:27:52.813928 15780 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:52.813308 15781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 85a6a7279939455ba36d3de1efc06a35. Latest consensus state: current_term: 1 leader_uuid: "85a6a7279939455ba36d3de1efc06a35" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "85a6a7279939455ba36d3de1efc06a35" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 38023 } } }
I20260806 09:27:52.814452 15781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:52.818158 15786 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:52.824244 15786 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:52.835853 15786 catalog_manager.cc:1347] Generated new cluster ID: 16a39525f64a49319a79108887e8c6c4
I20260806 09:27:52.835996 15786 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:52.859625 15786 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:52.861555 15786 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:52.879908 15786 catalog_manager.cc:5868] T 00000000000000000000000000000000 P 85a6a7279939455ba36d3de1efc06a35: Generated new TSK 0
I20260806 09:27:52.881107 15786 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:52.896618 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 15675
W20260806 09:27:52.897951 15756 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44491
W20260806 09:27:52.898837 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 15675: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:27:52.899010 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 15675
2026-08-06T09:27:52Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2520 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:53 dist-test-slave-h3ft krb5kdc[15805](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:53 dist-test-slave-h3ft krb5kdc[15805](info): set up 2 sockets
Aug 06 09:27:53 dist-test-slave-h3ft krb5kdc[15805](info): commencing operation
krb5kdc: starting...
W20260806 09:27:54.272346 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.322s user 0.000s sys 0.070s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:54 dist-test-slave-h3ft krb5kdc[15805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008474, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:54Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:27:54.687777 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:45279
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:35855
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:45279
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:55.116317 15822 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:55.116741 15822 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:55.116962 15822 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:55.125277 15822 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:55.125443 15822 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:55.125571 15822 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:55.125672 15822 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:55.125767 15822 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:55.137610 15822 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:35855
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:45279
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:45279
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:55.139642 15822 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:55.141359 15822 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:55.157198 15830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:55.157385 15837 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:55.157418 15833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:55.158810 15822 server_base.cc:1034] running on GCE node
I20260806 09:27:55.159871 15822 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:55.161626 15822 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:55.162891 15822 hybrid_clock.cc:648] HybridClock initialized: now 1786008475162796 us; error 76 us; skew 500 ppm
I20260806 09:27:55.163355 15822 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:55 dist-test-slave-h3ft krb5kdc[15805](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008475, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:55.167403 15822 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:27:55.169610 15822 webserver.cc:460] Webserver started at http://127.16.86.190:45795/ using document root <none> and password file <none>
I20260806 09:27:55.170478 15822 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:55.170663 15822 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:55.171106 15822 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:55.174714 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "c105d12e1758418391742d124475fefc"
format_stamp: "Formatted at 2026-08-06 09:27:55 on dist-test-slave-h3ft"
server_key: "843c3db8e508c4dd66d5ce43dd1d5ba5"
server_key_iv: "851dd1ca7ffe3ecefbc6dcb58b398a4d"
server_key_version: "encryptionkey@0"
I20260806 09:27:55.175743 15822 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "c105d12e1758418391742d124475fefc"
format_stamp: "Formatted at 2026-08-06 09:27:55 on dist-test-slave-h3ft"
server_key: "843c3db8e508c4dd66d5ce43dd1d5ba5"
server_key_iv: "851dd1ca7ffe3ecefbc6dcb58b398a4d"
server_key_version: "encryptionkey@0"
I20260806 09:27:55.182350 15822 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.003s
I20260806 09:27:55.186681 15846 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:55.188689 15822 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:27:55.188910 15822 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "c105d12e1758418391742d124475fefc"
format_stamp: "Formatted at 2026-08-06 09:27:55 on dist-test-slave-h3ft"
server_key: "843c3db8e508c4dd66d5ce43dd1d5ba5"
server_key_iv: "851dd1ca7ffe3ecefbc6dcb58b398a4d"
server_key_version: "encryptionkey@0"
I20260806 09:27:55.189152 15822 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:55.196177 15822 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:55.203714 15822 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:55.204087 15822 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:55.227365 15822 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:45279
I20260806 09:27:55.227466 15915 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:45279 every 8 connection(s)
I20260806 09:27:55.229180 15822 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:55.229605 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 15822
I20260806 09:27:55.229825 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:55.230576 16730 external_mini_cluster.cc:1428] Setting key ae161792cf22eef74cffe469f737718f
I20260806 09:27:55.235571 15916 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:55.241426 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:55.233103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41725 (local address 127.16.86.190:45279)
0806 09:27:55.234146 (+ 1043us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:55.234167 (+ 21us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:55.234235 (+ 68us) server_negotiation.cc:407] Connection header received
0806 09:27:55.236931 (+ 2696us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:55.236965 (+ 34us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:55.237493 (+ 528us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:55.238036 (+ 543us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:55.239187 (+ 1151us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:55.239197 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:55.239223 (+ 26us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:55.239252 (+ 29us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:55.239358 (+ 106us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:55.239387 (+ 29us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:55.239654 (+ 267us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:55.239807 (+ 153us) server_negotiation.cc:299] Negotiation successful
0806 09:27:55.240043 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":703,"thread_start_us":414,"threads_started":1}
I20260806 09:27:55.246295 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:55.250677 15916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc: Bootstrap starting.
I20260806 09:27:55.254907 15923 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:55.251818 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51699 (local address 127.16.86.190:45279)
0806 09:27:55.252045 (+ 227us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:55.252052 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:55.252462 (+ 410us) server_negotiation.cc:407] Connection header received
0806 09:27:55.252648 (+ 186us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:55.252654 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:55.252752 (+ 98us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:55.252931 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:55.254097 (+ 1166us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:55.254103 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:55.254108 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: PLAIN
0806 09:27:55.254128 (+ 20us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:55.254187 (+ 59us) server_negotiation.cc:1091] Received PLAIN auth, user=slave
0806 09:27:55.254210 (+ 23us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:55.254349 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:55.254523 (+ 174us) server_negotiation.cc:299] Negotiation successful
0806 09:27:55.254592 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20260806 09:27:55.255543 15916 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:55.257118 15916 log.cc:824] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:55.260627 15916 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc: No bootstrap required, opened a new log
I20260806 09:27:55.269586 15916 raft_consensus.cc:348] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c105d12e1758418391742d124475fefc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 45279 } }
I20260806 09:27:55.270071 15916 raft_consensus.cc:374] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260806 09:27:55.270213 15916 raft_consensus.cc:729] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c105d12e1758418391742d124475fefc, State: Initialized, Role: FOLLOWER
I20260806 09:27:55.271158 15916 consensus_queue.cc:260] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c105d12e1758418391742d124475fefc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 45279 } }
I20260806 09:27:55.271471 15916 raft_consensus.cc:388] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260806 09:27:55.271665 15916 raft_consensus.cc:482] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260806 09:27:55.271869 15916 raft_consensus.cc:3037] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:55.274459 15916 raft_consensus.cc:504] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c105d12e1758418391742d124475fefc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 45279 } }
I20260806 09:27:55.275174 15916 leader_election.cc:302] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c105d12e1758418391742d124475fefc; no voters:
I20260806 09:27:55.276046 15916 leader_election.cc:288] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [CANDIDATE]: Term 1 election: Requested vote from peers
I20260806 09:27:55.276260 15934 raft_consensus.cc:2781] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:55.277281 15934 raft_consensus.cc:686] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [term 1 LEADER]: Becoming Leader. State: Replica: c105d12e1758418391742d124475fefc, State: Running, Role: LEADER
I20260806 09:27:55.277990 15934 consensus_queue.cc:237] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c105d12e1758418391742d124475fefc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 45279 } }
I20260806 09:27:55.278779 15916 sys_catalog.cc:564] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:55.284453 15935 sys_catalog.cc:455] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c105d12e1758418391742d124475fefc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c105d12e1758418391742d124475fefc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 45279 } } }
I20260806 09:27:55.284902 15935 sys_catalog.cc:458] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:55.285724 15936 sys_catalog.cc:455] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [sys.catalog]: SysCatalogTable state changed. Reason: New leader c105d12e1758418391742d124475fefc. Latest consensus state: current_term: 1 leader_uuid: "c105d12e1758418391742d124475fefc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c105d12e1758418391742d124475fefc" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 45279 } } }
I20260806 09:27:55.285984 15943 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:55.286111 15936 sys_catalog.cc:458] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:55.292894 15943 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:27:55.304451 15943 catalog_manager.cc:1347] Generated new cluster ID: 134c9e19ba404af4b09e48a387cec05c
I20260806 09:27:55.304593 15943 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:55.328483 15943 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:55.329145 15943 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:55.348502 15943 catalog_manager.cc:5868] T 00000000000000000000000000000000 P c105d12e1758418391742d124475fefc: Generated new TSK 0
I20260806 09:27:55.350080 15943 catalog_manager.cc:1514] Initializing in-progress tserver states...
I20260806 09:27:55.358074 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 15822
W20260806 09:27:55.359651 15906 server_base.cc:1116] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41725
W20260806 09:27:55.360683 16730 external_mini_cluster.cc:1652] Unable to check leaks on /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 15822: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20260806 09:27:55.360893 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 15822
2026-08-06T09:27:55Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2460 ms)
[----------] 18 tests from AuthTokenIssuingTest (45212 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:27:55 dist-test-slave-h3ft krb5kdc[15960](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:27:55 dist-test-slave-h3ft krb5kdc[15960](info): set up 2 sockets
Aug 06 09:27:55 dist-test-slave-h3ft krb5kdc[15960](info): commencing operation
krb5kdc: starting...
W20260806 09:27:56.729492 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.317s user 0.003s sys 0.055s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:27:56 dist-test-slave-h3ft krb5kdc[15960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:27:57Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:27:57Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:27:57.147490 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:35111
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:39729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:35111,127.16.86.189:33653,127.16.86.188:36289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:27:57.548643 15979 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:27:57.549095 15979 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:57.549233 15979 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:57.549383 15979 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:57.556979 15979 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:57.557129 15979 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:57.557288 15979 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:57.557397 15979 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:57.557482 15979 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:57.567849 15979 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:39729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:35111,127.16.86.189:33653,127.16.86.188:36289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:35111
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:57.569196 15979 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:57.570514 15979 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:57.584894 15985 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:57.584822 15990 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:57.584774 15986 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:57.586462 15979 server_base.cc:1034] running on GCE node
I20260806 09:27:57.587653 15979 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:57.589488 15979 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:57.590791 15979 hybrid_clock.cc:648] HybridClock initialized: now 1786008477590691 us; error 54 us; skew 500 ppm
I20260806 09:27:57.591362 15979 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:57 dist-test-slave-h3ft krb5kdc[15960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008477, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:57.596771 15979 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:27:57.599248 15979 webserver.cc:460] Webserver started at http://127.16.86.190:43179/ using document root <none> and password file <none>
I20260806 09:27:57.600324 15979 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:57.600502 15979 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:57.600984 15979 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:57.605202 15979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "e7db430dacf04b929cc915527a3667f7"
format_stamp: "Formatted at 2026-08-06 09:27:57 on dist-test-slave-h3ft"
server_key: "406956f5739e0dddf7360044ba807a86"
server_key_iv: "4dad34110f743a8f95d5a60cfe6c970d"
server_key_version: "encryptionkey@0"
I20260806 09:27:57.606205 15979 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "e7db430dacf04b929cc915527a3667f7"
format_stamp: "Formatted at 2026-08-06 09:27:57 on dist-test-slave-h3ft"
server_key: "406956f5739e0dddf7360044ba807a86"
server_key_iv: "4dad34110f743a8f95d5a60cfe6c970d"
server_key_version: "encryptionkey@0"
I20260806 09:27:57.613224 15979 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260806 09:27:57.618284 15999 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:57.620163 15979 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260806 09:27:57.620409 15979 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "e7db430dacf04b929cc915527a3667f7"
format_stamp: "Formatted at 2026-08-06 09:27:57 on dist-test-slave-h3ft"
server_key: "406956f5739e0dddf7360044ba807a86"
server_key_iv: "4dad34110f743a8f95d5a60cfe6c970d"
server_key_version: "encryptionkey@0"
I20260806 09:27:57.620682 15979 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:57.637099 15979 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:57.642865 15979 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:57.643236 15979 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:57.667671 15979 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:35111
I20260806 09:27:57.667738 16069 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:35111 every 8 connection(s)
I20260806 09:27:57.669571 15979 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:27:57.670213 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 15979
I20260806 09:27:57.670475 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:27:57.671468 16730 external_mini_cluster.cc:1428] Setting key 6a437cdf59b427f7dd1c2a6e90aa50ac
I20260806 09:27:57.675066 16070 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:57.683480 16070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:27:57 dist-test-slave-h3ft krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008477, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
WARNING: no policy specified for kudu/127.16.86.189@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.189@KRBTEST.COM" created.
I20260806 09:27:57.717337 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:57.687507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35111 (local address 127.0.0.1:51398)
0806 09:27:57.688676 (+ 1169us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:57.688756 (+ 80us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:57.692136 (+ 3380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:57.693801 (+ 1665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:57.693841 (+ 40us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:57.694896 (+ 1055us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:57.695579 (+ 683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:57.695623 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:57.697681 (+ 2058us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:57.697690 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:57.700008 (+ 2318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:57.700032 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:57.701298 (+ 1266us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:57.701309 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:57.701620 (+ 311us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:57.702586 (+ 966us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:27:57.702632 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:57.706354 (+ 3722us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:57.711434 (+ 5080us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:57.711503 (+ 69us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:57.711522 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:57.712159 (+ 637us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:57.712936 (+ 777us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:57.712943 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:57.712947 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:57.713183 (+ 236us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:57.714176 (+ 993us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:57.714191 (+ 15us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:57.714457 (+ 266us) client_negotiation.cc:769] Sending connection context
0806 09:27:57.714756 (+ 299us) client_negotiation.cc:240] Negotiation successful
0806 09:27:57.715578 (+ 822us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":743,"mutex_wait_us":182,"thread_start_us":408,"threads_started":1}
I20260806 09:27:57.717357 16079 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:57.688977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51398 (local address 127.16.86.190:35111)
0806 09:27:57.689721 (+ 744us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:57.689727 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:57.689785 (+ 58us) server_negotiation.cc:407] Connection header received
0806 09:27:57.692814 (+ 3029us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:57.692827 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:57.693306 (+ 479us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:57.693639 (+ 333us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:57.695934 (+ 2295us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:57.697431 (+ 1497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:57.700254 (+ 2823us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:57.700959 (+ 705us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:57.701146 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:57.706761 (+ 5615us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:57.706773 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:57.706803 (+ 30us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:57.706851 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:57.711168 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:57.712377 (+ 1209us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:57.712385 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:57.712388 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:57.712712 (+ 324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:57.713313 (+ 601us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:57.713319 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:57.713322 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:57.714021 (+ 699us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:57.714280 (+ 259us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:57.714913 (+ 633us) server_negotiation.cc:299] Negotiation successful
0806 09:27:57.715389 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":546,"thread_start_us":379,"threads_started":1}
I20260806 09:27:57.721778 16070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:27:57.723414 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:57.722809 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:34540)
0806 09:27:57.723110 (+ 301us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:57.723254 (+ 144us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
W20260806 09:27:57.723804 16001 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
I20260806 09:27:57.726621 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:57.725954 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:34556)
0806 09:27:57.726218 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:57.726361 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260806 09:27:57.726935 16070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:33653: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.189.keytab.
Entry for principal kudu/127.16.86.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.189.keytab.
I20260806 09:27:57.764428 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.189:33653
--webserver_interface=127.16.86.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.189.keytab
--principal=kudu/127.16.86.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:39729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:35111,127.16.86.189:33653,127.16.86.188:36289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:27:57.776500 16070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } attempt: 1
I20260806 09:27:57.778242 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:57.777588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:34562)
0806 09:27:57.777881 (+ 293us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:57.778067 (+ 186us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":123}
I20260806 09:27:57.779513 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:57.779140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:34568)
0806 09:27:57.779289 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:57.779367 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
W20260806 09:27:57.779870 16070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:33653: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
I20260806 09:27:57.848204 16070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } attempt: 2
I20260806 09:27:57.849918 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:57.849175 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:34574)
0806 09:27:57.849543 (+ 368us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:57.849698 (+ 155us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":106}
I20260806 09:27:57.851302 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:57.850759 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:34590)
0806 09:27:57.850953 (+ 194us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:57.851103 (+ 150us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20260806 09:27:57.851580 16070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:33653: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
I20260806 09:27:57.943054 16070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } attempt: 3
I20260806 09:27:57.944681 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:57.944096 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:34606)
0806 09:27:57.944360 (+ 264us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:57.944480 (+ 120us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":107}
I20260806 09:27:57.946028 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:57.945610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:34622)
0806 09:27:57.945807 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:57.945882 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20260806 09:27:57.946216 16070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:33653: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
I20260806 09:27:58.089530 16070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } attempt: 4
I20260806 09:27:58.091290 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.090309 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:34628)
0806 09:27:58.090816 (+ 507us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.091041 (+ 225us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":248}
I20260806 09:27:58.092696 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.092186 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:34630)
0806 09:27:58.092383 (+ 197us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.092500 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 09:27:58.093008 16070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:33653: Network error: Client connection negotiation failed: client connection to 127.16.86.189:33653: connect: Connection refused (error 111)
W20260806 09:27:58.196740 16082 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:27:58.197165 16082 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:58.197252 16082 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:58.197347 16082 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:58.205351 16082 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:58.205483 16082 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:58.205549 16082 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:58.205600 16082 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:58.205685 16082 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:58.218169 16082 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:39729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:35111,127.16.86.189:33653,127.16.86.188:36289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.189.keytab
--principal=kudu/127.16.86.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.189:33653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.86.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:58.219816 16082 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:58.221249 16082 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:58.236056 16087 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:58.236191 16091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:58.236191 16088 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:58.237447 16082 server_base.cc:1034] running on GCE node
I20260806 09:27:58.238572 16082 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:58.240527 16082 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:58.241816 16082 hybrid_clock.cc:648] HybridClock initialized: now 1786008478241742 us; error 40 us; skew 500 ppm
I20260806 09:27:58.242250 16082 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:58 dist-test-slave-h3ft krb5kdc[15960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:58.246449 16082 init.cc:377] Logged in from keytab as kudu/127.16.86.189@KRBTEST.COM (short username kudu)
I20260806 09:27:58.248392 16082 webserver.cc:460] Webserver started at http://127.16.86.189:44237/ using document root <none> and password file <none>
I20260806 09:27:58.249279 16082 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:58.249435 16082 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:58.249827 16082 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:58.253538 16082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/data/instance:
uuid: "af9e9c6ee7464f39bb0bea6437f1990f"
format_stamp: "Formatted at 2026-08-06 09:27:58 on dist-test-slave-h3ft"
server_key: "f21e75ecfcb3893903b1952d1ee5374a"
server_key_iv: "3d14d1bb9e66ce5727f7ba3f069f458c"
server_key_version: "encryptionkey@0"
I20260806 09:27:58.254515 16082 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/wal/instance:
uuid: "af9e9c6ee7464f39bb0bea6437f1990f"
format_stamp: "Formatted at 2026-08-06 09:27:58 on dist-test-slave-h3ft"
server_key: "f21e75ecfcb3893903b1952d1ee5374a"
server_key_iv: "3d14d1bb9e66ce5727f7ba3f069f458c"
server_key_version: "encryptionkey@0"
I20260806 09:27:58.261109 16082 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260806 09:27:58.265444 16100 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:58.267148 16082 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:27:58.267365 16082 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/wal
uuid: "af9e9c6ee7464f39bb0bea6437f1990f"
format_stamp: "Formatted at 2026-08-06 09:27:58 on dist-test-slave-h3ft"
server_key: "f21e75ecfcb3893903b1952d1ee5374a"
server_key_iv: "3d14d1bb9e66ce5727f7ba3f069f458c"
server_key_version: "encryptionkey@0"
I20260806 09:27:58.267647 16082 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:58.288136 16082 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:58.293530 16082 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:58.293869 16082 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:58.317132 16082 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.189:33653
I20260806 09:27:58.317178 16181 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.189:33653 every 8 connection(s)
I20260806 09:27:58.318869 16082 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
I20260806 09:27:58.319454 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 16082
I20260806 09:27:58.319708 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-1/wal/instance
I20260806 09:27:58.320362 16730 external_mini_cluster.cc:1428] Setting key d8345fc6d699a313299bbf0734cf1d60
I20260806 09:27:58.324420 16182 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:58.335198 16182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:27:58 dist-test-slave-h3ft krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.189@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:27:58.366178 16187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.338912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51414 (local address 127.16.86.190:35111)
0806 09:27:58.339631 (+ 719us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:58.339638 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:58.340936 (+ 1298us) server_negotiation.cc:407] Connection header received
0806 09:27:58.344560 (+ 3624us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:58.344565 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:58.344659 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:58.344767 (+ 108us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:58.347522 (+ 2755us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.348356 (+ 834us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.351185 (+ 2829us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.351650 (+ 465us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.351773 (+ 123us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:58.357274 (+ 5501us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:58.357295 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:58.357303 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:58.357383 (+ 80us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:58.361841 (+ 4458us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:58.362967 (+ 1126us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:58.362977 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:58.362983 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:58.363206 (+ 223us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:58.363773 (+ 567us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:58.363778 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:58.363782 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:58.364272 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:58.364456 (+ 184us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:58.365332 (+ 876us) server_negotiation.cc:299] Negotiation successful
0806 09:27:58.365628 (+ 296us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":342,"threads_started":1}
WARNING: no policy specified for kudu/127.16.86.188@KRBTEST.COM; defaulting to no policy
I20260806 09:27:58.366904 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.339476 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35111 (local address 127.0.0.1:51414)
0806 09:27:58.340667 (+ 1191us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.340751 (+ 84us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:58.344119 (+ 3368us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:58.345138 (+ 1019us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:58.345154 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:58.346196 (+ 1042us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:58.347243 (+ 1047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:58.347276 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.348559 (+ 1283us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.348568 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:58.350954 (+ 2386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:58.350974 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.351971 (+ 997us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.351984 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:58.352378 (+ 394us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:58.353363 (+ 985us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:27:58.353403 (+ 40us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:58.356865 (+ 3462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:58.362087 (+ 5222us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:58.362097 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:58.362118 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:58.362765 (+ 647us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:58.363393 (+ 628us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:58.363397 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:58.363400 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:58.363610 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:58.364424 (+ 814us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:58.364430 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:58.364889 (+ 459us) client_negotiation.cc:769] Sending connection context
0806 09:27:58.365160 (+ 271us) client_negotiation.cc:240] Negotiation successful
0806 09:27:58.365607 (+ 447us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":744,"thread_start_us":426,"threads_started":1}
I20260806 09:27:58.369401 16182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.188@KRBTEST.COM" created.
Aug 06 09:27:58 dist-test-slave-h3ft krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.189@KRBTEST.COM for kudu/127.16.86.189@KRBTEST.COM
I20260806 09:27:58.384929 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.370408 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:34634)
0806 09:27:58.370650 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.370677 (+ 27us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:58.370791 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:58.372052 (+ 1261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:58.372056 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:58.372658 (+ 602us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:58.372801 (+ 143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:58.372809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.374248 (+ 1439us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.374256 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:58.375161 (+ 905us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:58.375172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.376007 (+ 835us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.376012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:58.376269 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:58.376960 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:27:58.376987 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:58.379100 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:58.382357 (+ 3257us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:58.382363 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:58.382367 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:58.382722 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:58.383154 (+ 432us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:58.383158 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:58.383162 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:58.383306 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:58.383905 (+ 599us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:58.383915 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:58.384051 (+ 136us) client_negotiation.cc:769] Sending connection context
0806 09:27:58.384209 (+ 158us) client_negotiation.cc:240] Negotiation successful
0806 09:27:58.384498 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":73,"mutex_wait_us":44}
I20260806 09:27:58.384976 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.370743 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34634 (local address 127.16.86.189:33653)
0806 09:27:58.371594 (+ 851us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:58.371600 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:58.371635 (+ 35us) server_negotiation.cc:407] Connection header received
0806 09:27:58.371680 (+ 45us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:58.371685 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:58.371898 (+ 213us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:58.372062 (+ 164us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:58.372982 (+ 920us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.374110 (+ 1128us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.375302 (+ 1192us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.375852 (+ 550us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.375992 (+ 140us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:58.379315 (+ 3323us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:58.379325 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:58.379339 (+ 14us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:58.379370 (+ 31us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:58.382209 (+ 2839us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:58.382862 (+ 653us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:58.382869 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:58.382871 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:58.382985 (+ 114us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:58.383412 (+ 427us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:58.383419 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:58.383422 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:58.383782 (+ 360us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:58.384009 (+ 227us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:58.384350 (+ 341us) server_negotiation.cc:299] Negotiation successful
0806 09:27:58.384602 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":700,"thread_start_us":528,"threads_started":1}
I20260806 09:27:58.388535 16182 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:27:58.389693 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.389218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:38812)
0806 09:27:58.389416 (+ 198us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.389526 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20260806 09:27:58.389952 16104 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
I20260806 09:27:58.391690 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.391319 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:38828)
0806 09:27:58.391502 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.391580 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
W20260806 09:27:58.391938 16182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:36289: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
I20260806 09:27:58.392246 16070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } attempt: 5
Aug 06 09:27:58 dist-test-slave-h3ft krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008477, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for kudu/127.16.86.189@KRBTEST.COM
I20260806 09:27:58.407454 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.392989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:34648)
0806 09:27:58.393241 (+ 252us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.393265 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:58.393451 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:58.393830 (+ 379us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:58.393836 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:58.394305 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:58.394498 (+ 193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:58.394507 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.395708 (+ 1201us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.395717 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:58.396795 (+ 1078us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:58.396809 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.397605 (+ 796us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.397612 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:58.397862 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:58.398646 (+ 784us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:27:58.398667 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:58.401129 (+ 2462us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:58.404644 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:58.404650 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:58.404655 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:58.405094 (+ 439us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:58.405505 (+ 411us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:58.405511 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:58.405514 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:58.405668 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:58.406272 (+ 604us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:58.406277 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:58.406459 (+ 182us) client_negotiation.cc:769] Sending connection context
0806 09:27:58.406651 (+ 192us) client_negotiation.cc:240] Negotiation successful
0806 09:27:58.406992 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":97}
I20260806 09:27:58.407449 16189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.393085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34648 (local address 127.16.86.189:33653)
0806 09:27:58.393246 (+ 161us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:58.393254 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:58.393367 (+ 113us) server_negotiation.cc:407] Connection header received
0806 09:27:58.393617 (+ 250us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:58.393622 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:58.393683 (+ 61us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:58.393795 (+ 112us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:58.394660 (+ 865us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.395500 (+ 840us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.396970 (+ 1470us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.397431 (+ 461us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.397591 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:58.401349 (+ 3758us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:58.401355 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:58.401358 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:58.401390 (+ 32us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:58.404469 (+ 3079us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:58.405225 (+ 756us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:58.405233 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:58.405237 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:58.405376 (+ 139us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:58.405795 (+ 419us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:58.405802 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:58.405805 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:58.406123 (+ 318us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:58.406298 (+ 175us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:58.406771 (+ 473us) server_negotiation.cc:299] Negotiation successful
0806 09:27:58.407073 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20260806 09:27:58.408777 16070 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:27:58.410162 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.409483 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:38842)
0806 09:27:58.409742 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.409891 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:27:58.411484 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.410989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:38848)
0806 09:27:58.411231 (+ 242us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.411352 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
W20260806 09:27:58.411761 16070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:36289: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.188.keytab.
Entry for principal kudu/127.16.86.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.188.keytab.
W20260806 09:27:58.423545 16730 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.86.188: real 0.103s user 0.000s sys 0.037s
I20260806 09:27:58.424547 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.188:36289
--webserver_interface=127.16.86.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.188.keytab
--principal=kudu/127.16.86.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:39729
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:35111,127.16.86.189:33653,127.16.86.188:36289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:27:58.441290 16182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } attempt: 1
I20260806 09:27:58.442744 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.442115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:38854)
0806 09:27:58.442340 (+ 225us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.442510 (+ 170us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
I20260806 09:27:58.444482 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.443892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:38868)
0806 09:27:58.444189 (+ 297us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.444314 (+ 125us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":101}
W20260806 09:27:58.444808 16182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:36289: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
I20260806 09:27:58.463142 16070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } attempt: 1
I20260806 09:27:58.464771 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.464170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:38880)
0806 09:27:58.464443 (+ 273us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.464564 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 09:27:58.465890 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.465554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:38890)
0806 09:27:58.465703 (+ 149us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.465786 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 09:27:58.466221 16070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:36289: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
I20260806 09:27:58.513082 16182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } attempt: 2
I20260806 09:27:58.514635 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.513989 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47200)
0806 09:27:58.514276 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.514417 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
I20260806 09:27:58.515909 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.515473 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47202)
0806 09:27:58.515644 (+ 171us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.515741 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 09:27:58.516263 16182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:36289: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
I20260806 09:27:58.534508 16070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } attempt: 2
I20260806 09:27:58.536276 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.535541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47210)
0806 09:27:58.535878 (+ 337us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.536007 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260806 09:27:58.537434 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.537099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47226)
0806 09:27:58.537245 (+ 146us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.537331 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 09:27:58.537637 16070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:36289: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
I20260806 09:27:58.607625 16182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } attempt: 3
I20260806 09:27:58.609488 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.608844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47230)
0806 09:27:58.609125 (+ 281us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.609286 (+ 161us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20260806 09:27:58.610867 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.610388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47246)
0806 09:27:58.610657 (+ 269us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.610741 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
W20260806 09:27:58.611222 16182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:36289: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
I20260806 09:27:58.643973 16070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } attempt: 3
I20260806 09:27:58.645977 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.645122 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47256)
0806 09:27:58.645482 (+ 360us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.645663 (+ 181us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20260806 09:27:58.647421 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.646862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47272)
0806 09:27:58.647149 (+ 287us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.647270 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":120}
W20260806 09:27:58.647712 16070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:36289: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
I20260806 09:27:58.754470 16182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } attempt: 4
I20260806 09:27:58.756405 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.755595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47276)
0806 09:27:58.755984 (+ 389us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.756157 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20260806 09:27:58.757783 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.757395 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47290)
0806 09:27:58.757568 (+ 173us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.757659 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
W20260806 09:27:58.758090 16182 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:36289: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
I20260806 09:27:58.824973 16070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } attempt: 4
I20260806 09:27:58.826596 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.825964 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47298)
0806 09:27:58.826258 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.826395 (+ 137us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":116}
I20260806 09:27:58.828085 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.827574 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47306)
0806 09:27:58.827773 (+ 199us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.827880 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
W20260806 09:27:58.828460 16070 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:36289: Network error: Client connection negotiation failed: client connection to 127.16.86.188:36289: connect: Connection refused (error 111)
W20260806 09:27:58.829909 16192 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:27:58.830299 16192 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:27:58.830416 16192 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:27:58.830554 16192 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:27:58.838109 16192 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:27:58.838203 16192 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:27:58.838295 16192 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:27:58.838376 16192 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:27:58.838472 16192 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:27:58.853964 16192 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:39729
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:35111,127.16.86.189:33653,127.16.86.188:36289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.188.keytab
--principal=kudu/127.16.86.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.188:36289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.86.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:27:58.855465 16192 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:27:58.856841 16192 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:27:58.871209 16199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:58.871166 16198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:27:58.871260 16203 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:27:58.872643 16192 server_base.cc:1034] running on GCE node
I20260806 09:27:58.873569 16192 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:27:58.875372 16192 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:27:58.876693 16192 hybrid_clock.cc:648] HybridClock initialized: now 1786008478876561 us; error 41 us; skew 500 ppm
I20260806 09:27:58.877131 16192 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:27:58 dist-test-slave-h3ft krb5kdc[15960](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:27:58.881109 16192 init.cc:377] Logged in from keytab as kudu/127.16.86.188@KRBTEST.COM (short username kudu)
I20260806 09:27:58.882831 16192 webserver.cc:460] Webserver started at http://127.16.86.188:46173/ using document root <none> and password file <none>
I20260806 09:27:58.883633 16192 fs_manager.cc:359] Metadata directory not provided
I20260806 09:27:58.883731 16192 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:27:58.884004 16192 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:27:58.887115 16192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/data/instance:
uuid: "1567baa76dc2425ba644ab77a2afb919"
format_stamp: "Formatted at 2026-08-06 09:27:58 on dist-test-slave-h3ft"
server_key: "4aff7af9bee35900e5692017dcdedb49"
server_key_iv: "aaded1739c675ecd2a9fc3ea17bd9d5d"
server_key_version: "encryptionkey@0"
I20260806 09:27:58.887934 16192 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/wal/instance:
uuid: "1567baa76dc2425ba644ab77a2afb919"
format_stamp: "Formatted at 2026-08-06 09:27:58 on dist-test-slave-h3ft"
server_key: "4aff7af9bee35900e5692017dcdedb49"
server_key_iv: "aaded1739c675ecd2a9fc3ea17bd9d5d"
server_key_version: "encryptionkey@0"
I20260806 09:27:58.893811 16192 fs_manager.cc:691] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260806 09:27:58.898020 16213 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:27:58.899808 16192 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260806 09:27:58.900038 16192 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/wal
uuid: "1567baa76dc2425ba644ab77a2afb919"
format_stamp: "Formatted at 2026-08-06 09:27:58 on dist-test-slave-h3ft"
server_key: "4aff7af9bee35900e5692017dcdedb49"
server_key_iv: "aaded1739c675ecd2a9fc3ea17bd9d5d"
server_key_version: "encryptionkey@0"
I20260806 09:27:58.900332 16192 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:27:58.926235 16192 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:27:58.930819 16192 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:27:58.931186 16192 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:27:58.953156 16192 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.188:36289
I20260806 09:27:58.953218 16274 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.188:36289 every 8 connection(s)
I20260806 09:27:58.954684 16192 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
I20260806 09:27:58.959637 16276 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:27:58.960103 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 16192
I20260806 09:27:58.960350 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1786008061536593-16730-0/minicluster-data/master-2/wal/instance
I20260806 09:27:58.961043 16730 external_mini_cluster.cc:1428] Setting key 60d550d394c9732acf430a3df6f4f163
I20260806 09:27:58.970965 16276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:27:58 dist-test-slave-h3ft krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.188@KRBTEST.COM
Aug 06 09:27:58 dist-test-slave-h3ft krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.188@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:27:58.991307 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.963721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47308 (local address 127.16.86.188:36289)
0806 09:27:58.964720 (+ 999us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:58.964736 (+ 16us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:58.964785 (+ 49us) server_negotiation.cc:407] Connection header received
0806 09:27:58.967686 (+ 2901us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:58.967716 (+ 30us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:58.968245 (+ 529us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:58.968721 (+ 476us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:58.970379 (+ 1658us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.971914 (+ 1535us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.973459 (+ 1545us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.974502 (+ 1043us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.974723 (+ 221us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:58.981114 (+ 6391us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:58.981128 (+ 14us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:58.981140 (+ 12us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:58.981190 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:58.985364 (+ 4174us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:58.986204 (+ 840us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:58.986220 (+ 16us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:58.986227 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:58.986417 (+ 190us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:58.986891 (+ 474us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:58.986900 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:58.986907 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:58.987892 (+ 985us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:58.988193 (+ 301us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:58.988763 (+ 570us) server_negotiation.cc:299] Negotiation successful
0806 09:27:58.989403 (+ 640us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":718,"thread_start_us":417,"threads_started":1}
I20260806 09:27:58.995945 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.974548 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:35111 (local address 127.0.0.1:38436)
0806 09:27:58.977382 (+ 2834us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.977440 (+ 58us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:58.977714 (+ 274us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:58.978218 (+ 504us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:58.978229 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:58.979353 (+ 1124us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:58.979674 (+ 321us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:58.979685 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.980981 (+ 1296us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.980989 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:58.982766 (+ 1777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:58.982783 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.983809 (+ 1026us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.983820 (+ 11us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:58.984058 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:58.984804 (+ 746us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:27:58.984824 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:58.987730 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:58.992317 (+ 4587us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:58.992327 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:58.992335 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:58.993052 (+ 717us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:58.993615 (+ 563us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:58.993620 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:58.993623 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:58.993772 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:58.994582 (+ 810us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:58.994593 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:58.994897 (+ 304us) client_negotiation.cc:769] Sending connection context
0806 09:27:58.995166 (+ 269us) client_negotiation.cc:240] Negotiation successful
0806 09:27:58.995442 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2491,"thread_start_us":1460,"threads_started":1}
I20260806 09:27:58.996035 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.974532 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38436 (local address 127.16.86.190:35111)
0806 09:27:58.976963 (+ 2431us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:58.976969 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:58.977591 (+ 622us) server_negotiation.cc:407] Connection header received
0806 09:27:58.977885 (+ 294us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:58.977889 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:58.977960 (+ 71us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:58.978061 (+ 101us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:58.979945 (+ 1884us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.980780 (+ 835us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.983031 (+ 2251us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.983542 (+ 511us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:58.983707 (+ 165us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:58.988053 (+ 4346us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:58.988061 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:58.988066 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:58.988149 (+ 83us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:58.992154 (+ 4005us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:58.993250 (+ 1096us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:58.993255 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:58.993259 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:58.993451 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:58.993964 (+ 513us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:58.993969 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:58.993972 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:58.994375 (+ 403us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:58.994554 (+ 179us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:58.995357 (+ 803us) server_negotiation.cc:299] Negotiation successful
0806 09:27:58.995630 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2312,"thread_start_us":2172,"threads_started":1}
I20260806 09:27:58.998032 16276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:27:59 dist-test-slave-h3ft krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
Aug 06 09:27:59 dist-test-slave-h3ft krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.188@KRBTEST.COM for kudu/127.16.86.189@KRBTEST.COM
I20260806 09:27:59.014787 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.996093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38446 (local address 127.16.86.190:35111)
0806 09:27:58.996337 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:58.996346 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:58.996373 (+ 27us) server_negotiation.cc:407] Connection header received
0806 09:27:58.996671 (+ 298us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:58.996681 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:58.996795 (+ 114us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:58.996875 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:58.998462 (+ 1587us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:58.999454 (+ 992us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.001310 (+ 1856us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.001796 (+ 486us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.001906 (+ 110us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.007880 (+ 5974us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:59.007892 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:59.007901 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:59.007957 (+ 56us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:59.011513 (+ 3556us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.012228 (+ 715us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.012234 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.012238 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.012388 (+ 150us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.012966 (+ 578us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.012972 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.012975 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.013341 (+ 366us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:59.013686 (+ 345us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:59.014078 (+ 392us) server_negotiation.cc:299] Negotiation successful
0806 09:27:59.014336 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":106,"thread_start_us":424,"threads_started":1}
I20260806 09:27:59.015439 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.998959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:33653 (local address 127.0.0.1:46996)
0806 09:27:58.999164 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:58.999182 (+ 18us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:58.999286 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:59.000250 (+ 964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:59.000255 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:59.000724 (+ 469us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:59.000962 (+ 238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:59.000975 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.002374 (+ 1399us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.002378 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:59.003694 (+ 1316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:59.003710 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.004891 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.004896 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:59.005097 (+ 201us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.005915 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:27:59.005944 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:59.008909 (+ 2965us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:59.012508 (+ 3599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:59.012514 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:59.012520 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:59.013003 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:59.013455 (+ 452us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:59.013459 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:59.013463 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:59.013660 (+ 197us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:59.014356 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:59.014361 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:59.014554 (+ 193us) client_negotiation.cc:769] Sending connection context
0806 09:27:59.014716 (+ 162us) client_negotiation.cc:240] Negotiation successful
0806 09:27:59.015086 (+ 370us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 09:27:59.015733 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:58.999137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46996 (local address 127.16.86.189:33653)
0806 09:27:58.999772 (+ 635us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:58.999778 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:58.999796 (+ 18us) server_negotiation.cc:407] Connection header received
0806 09:27:58.999857 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:58.999861 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:58.999944 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.000228 (+ 284us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.001192 (+ 964us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.002207 (+ 1015us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.003868 (+ 1661us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.004694 (+ 826us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.004873 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.009186 (+ 4313us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:59.009192 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:59.009195 (+ 3us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:59.009239 (+ 44us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:59.012337 (+ 3098us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.013166 (+ 829us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.013170 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.013173 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.013324 (+ 151us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.013813 (+ 489us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.013819 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.013822 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.014194 (+ 372us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:59.014367 (+ 173us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:59.014895 (+ 528us) server_negotiation.cc:299] Negotiation successful
0806 09:27:59.015205 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":493,"thread_start_us":318,"threads_started":1}
I20260806 09:27:59.016863 16276 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:27:59 dist-test-slave-h3ft krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.188@KRBTEST.COM for kudu/127.16.86.188@KRBTEST.COM
Aug 06 09:27:59 dist-test-slave-h3ft krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008476, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.189@KRBTEST.COM
I20260806 09:27:59.035897 16282 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:59.017778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47324)
0806 09:27:59.018025 (+ 247us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:59.018039 (+ 14us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:59.018115 (+ 76us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:59.019567 (+ 1452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:59.019572 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:59.020176 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:59.020377 (+ 201us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:59.020389 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.021867 (+ 1478us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.021871 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:59.023457 (+ 1586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:59.023471 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.024496 (+ 1025us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.024501 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:59.024766 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.025460 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:27:59.025481 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:59.028219 (+ 2738us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:59.032334 (+ 4115us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:59.032342 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:59.032350 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:59.032915 (+ 565us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:59.033458 (+ 543us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:59.033462 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:59.033465 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:59.033682 (+ 217us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:59.034409 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:59.034413 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:59.034725 (+ 312us) client_negotiation.cc:769] Sending connection context
0806 09:27:59.034946 (+ 221us) client_negotiation.cc:240] Negotiation successful
0806 09:27:59.035369 (+ 423us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":147}
I20260806 09:27:59.036149 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:59.017844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47324 (local address 127.16.86.188:36289)
0806 09:27:59.019230 (+ 1386us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.019239 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.019260 (+ 21us) server_negotiation.cc:407] Connection header received
0806 09:27:59.019319 (+ 59us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.019323 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.019404 (+ 81us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.019493 (+ 89us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.020534 (+ 1041us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.021674 (+ 1140us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.023620 (+ 1946us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.024278 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.024461 (+ 183us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.028470 (+ 4009us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:59.028478 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:59.028483 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:59.028535 (+ 52us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:59.032171 (+ 3636us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.033095 (+ 924us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.033104 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.033107 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.033291 (+ 184us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.033824 (+ 533us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.033831 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.033837 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.034260 (+ 423us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:59.034473 (+ 213us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:59.035137 (+ 664us) server_negotiation.cc:299] Negotiation successful
0806 09:27:59.035626 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":140,"server-negotiator.queue_time_us":1248}
I20260806 09:27:59.036690 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:59.017122 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47008 (local address 127.16.86.189:33653)
0806 09:27:59.017314 (+ 192us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.017319 (+ 5us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.017406 (+ 87us) server_negotiation.cc:407] Connection header received
0806 09:27:59.017623 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.017631 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.017721 (+ 90us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.017832 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.019355 (+ 1523us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.020527 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.022290 (+ 1763us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.023052 (+ 762us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.023237 (+ 185us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.029567 (+ 6330us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:59.029575 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:59.029579 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:59.029627 (+ 48us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:59.033001 (+ 3374us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.033889 (+ 888us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.033893 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.033895 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.034041 (+ 146us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.034533 (+ 492us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.034539 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.034543 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.034963 (+ 420us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:59.035320 (+ 357us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:59.035664 (+ 344us) server_negotiation.cc:299] Negotiation successful
0806 09:27:59.035951 (+ 287us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20260806 09:27:59.042677 16276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919: Bootstrap starting.
I20260806 09:27:59.043349 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:27:59.047056 16276 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:59.048339 16276 log.cc:824] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:59.047513 16039 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "e7db430dacf04b929cc915527a3667f7"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:38446
I20260806 09:27:59.052243 16276 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919: No bootstrap required, opened a new log
I20260806 09:27:59.057371 16182 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } attempt: 5
I20260806 09:27:59.060664 16276 raft_consensus.cc:348] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } }
I20260806 09:27:59.061065 16276 raft_consensus.cc:729] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1567baa76dc2425ba644ab77a2afb919, State: Initialized, Role: FOLLOWER
I20260806 09:27:59.061695 16276 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } }
I20260806 09:27:59.063030 16286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } } }
I20260806 09:27:59.063777 16286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:27:59.064417 16276 sys_catalog.cc:564] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [sys.catalog]: configured and running, proceeding with master startup.
Aug 06 09:27:59 dist-test-slave-h3ft krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008478, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.189@KRBTEST.COM for kudu/127.16.86.188@KRBTEST.COM
I20260806 09:27:59.078136 16188 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:59.058303 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47336)
0806 09:27:59.058628 (+ 325us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:59.058650 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:59.058775 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:59.059493 (+ 718us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:59.059499 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:59.060013 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:59.060265 (+ 252us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:59.060279 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.061752 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.061757 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:59.063212 (+ 1455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:59.063230 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.064611 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.064619 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:59.064864 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.065629 (+ 765us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:27:59.065649 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:59.068815 (+ 3166us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:59.074147 (+ 5332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:59.074154 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:59.074158 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:59.074864 (+ 706us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:59.075897 (+ 1033us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:59.075904 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:59.075908 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:59.076166 (+ 258us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:59.076979 (+ 813us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:59.076983 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:59.077183 (+ 200us) client_negotiation.cc:769] Sending connection context
0806 09:27:59.077351 (+ 168us) client_negotiation.cc:240] Negotiation successful
0806 09:27:59.077746 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":187}
W20260806 09:27:59.079181 16303 catalog_manager.cc:1558] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260806 09:27:59.079375 16303 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:27:59.079694 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:59.058684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47336 (local address 127.16.86.188:36289)
0806 09:27:59.058963 (+ 279us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.058972 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.059062 (+ 90us) server_negotiation.cc:407] Connection header received
0806 09:27:59.059156 (+ 94us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.059162 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.059262 (+ 100us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.059441 (+ 179us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.060445 (+ 1004us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.061563 (+ 1118us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.063433 (+ 1870us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.064325 (+ 892us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.064485 (+ 160us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.069092 (+ 4607us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:59.069099 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:59.069105 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:59.069163 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:59.073952 (+ 4789us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.075396 (+ 1444us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.075405 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.075412 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.075684 (+ 272us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.076345 (+ 661us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.076350 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.076353 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.076823 (+ 470us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:59.078767 (+ 1944us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:59.078919 (+ 152us) server_negotiation.cc:299] Negotiation successful
0806 09:27:59.079254 (+ 335us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 09:27:59.086191 16182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f: Bootstrap starting.
I20260806 09:27:59.075457 16192 master_runner.cc:186] Error getting master registration for 127.16.86.190:35111: OK, instance_id { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" instance_seqno: 1786008477660502 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:27:59.087181 16192 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 09:27:59.089687 16182 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:59.091109 16182 log.cc:824] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:59.094986 16182 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f: No bootstrap required, opened a new log
I20260806 09:27:59.105734 16070 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } attempt: 5
I20260806 09:27:59.106491 16182 raft_consensus.cc:348] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } }
I20260806 09:27:59.106946 16182 raft_consensus.cc:729] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af9e9c6ee7464f39bb0bea6437f1990f, State: Initialized, Role: FOLLOWER
I20260806 09:27:59.107560 16182 consensus_queue.cc:260] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } }
I20260806 09:27:59.109408 16310 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } } }
I20260806 09:27:59.110430 16182 sys_catalog.cc:564] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:59.110729 16310 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [sys.catalog]: This master's current role is: FOLLOWER
Aug 06 09:27:59 dist-test-slave-h3ft krb5kdc[15960](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008477, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for kudu/127.16.86.188@KRBTEST.COM
I20260806 09:27:59.130717 16078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:59.106701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:36289 (local address 127.0.0.1:47342)
0806 09:27:59.107076 (+ 375us) negotiation.cc:107] Waiting for socket to connect
0806 09:27:59.107099 (+ 23us) client_negotiation.cc:174] Beginning negotiation
0806 09:27:59.107238 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:27:59.107804 (+ 566us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:27:59.107810 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:27:59.108441 (+ 631us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:27:59.108697 (+ 256us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:59.108711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.110289 (+ 1578us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.110294 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:59.111946 (+ 1652us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:27:59.111979 (+ 33us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.113786 (+ 1807us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.113795 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:27:59.114088 (+ 293us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.115067 (+ 979us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:27:59.115104 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:27:59.119564 (+ 4460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:27:59.126688 (+ 7124us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:59.126698 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:59.126706 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:59.127437 (+ 731us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:59.128064 (+ 627us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:27:59.128072 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:27:59.128077 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:27:59.128401 (+ 324us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:27:59.129318 (+ 917us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:27:59.129326 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:27:59.129570 (+ 244us) client_negotiation.cc:769] Sending connection context
0806 09:27:59.129755 (+ 185us) client_negotiation.cc:240] Negotiation successful
0806 09:27:59.130188 (+ 433us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189}
I20260806 09:27:59.131083 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:59.106837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47342 (local address 127.16.86.188:36289)
0806 09:27:59.107112 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.107120 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.107154 (+ 34us) server_negotiation.cc:407] Connection header received
0806 09:27:59.107425 (+ 271us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.107432 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.107538 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.107713 (+ 175us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.108938 (+ 1225us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.110101 (+ 1163us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.112187 (+ 2086us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.113275 (+ 1088us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.113451 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.120014 (+ 6563us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:59.120026 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:59.120035 (+ 9us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:59.120112 (+ 77us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:59.126411 (+ 6299us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.127669 (+ 1258us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.127674 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.127680 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.127872 (+ 192us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.128622 (+ 750us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.128631 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.128637 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.129127 (+ 490us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:59.129360 (+ 233us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:59.129989 (+ 629us) server_negotiation.cc:299] Negotiation successful
0806 09:27:59.130388 (+ 399us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20260806 09:27:59.140291 16070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7: Bootstrap starting.
I20260806 09:27:59.145895 16070 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7: Neither blocks nor log segments found. Creating new log.
I20260806 09:27:59.147737 16070 log.cc:824] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7: Log is configured to *not* fsync() on all Append() calls
I20260806 09:27:59.146723 16082 master_runner.cc:186] Error getting master registration for 127.16.86.190:35111: OK, instance_id { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" instance_seqno: 1786008477660502 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:27:59.168761 16070 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7: No bootstrap required, opened a new log
W20260806 09:27:59.167222 16326 catalog_manager.cc:1558] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260806 09:27:59.175376 16082 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20260806 09:27:59.175323 16326 catalog_manager.cc:873] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260806 09:27:59.183292 16070 raft_consensus.cc:348] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } }
W20260806 09:27:59.183915 16223 tablet.cc:2369] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:27:59.184196 16070 raft_consensus.cc:729] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e7db430dacf04b929cc915527a3667f7, State: Initialized, Role: FOLLOWER
I20260806 09:27:59.185194 16070 consensus_queue.cc:260] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } }
I20260806 09:27:59.186874 16331 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } } }
I20260806 09:27:59.187547 16331 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:27:59.189708 16039 raft_consensus.cc:482] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 09:27:59.189946 16039 raft_consensus.cc:3037] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:59.190233 16070 sys_catalog.cc:564] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:27:59.193928 16039 raft_consensus.cc:504] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } }
I20260806 09:27:59.196833 16144 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7db430dacf04b929cc915527a3667f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "af9e9c6ee7464f39bb0bea6437f1990f"
I20260806 09:27:59.197403 16144 raft_consensus.cc:3037] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:59.201597 16144 raft_consensus.cc:2454] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7db430dacf04b929cc915527a3667f7 in term 1.
I20260806 09:27:59.202430 16001 leader_election.cc:302] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: af9e9c6ee7464f39bb0bea6437f1990f, e7db430dacf04b929cc915527a3667f7; no voters:
I20260806 09:27:59.203078 16331 raft_consensus.cc:2781] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:27:59.207599 16247 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e7db430dacf04b929cc915527a3667f7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "1567baa76dc2425ba644ab77a2afb919"
I20260806 09:27:59.208218 16247 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:27:59.212430 16247 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e7db430dacf04b929cc915527a3667f7 in term 1.
I20260806 09:27:59.213477 16039 leader_election.cc:288] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [CANDIDATE]: Term 1 election: Requested vote from peers af9e9c6ee7464f39bb0bea6437f1990f (127.16.86.189:33653), 1567baa76dc2425ba644ab77a2afb919 (127.16.86.188:36289)
I20260806 09:27:59.235153 16331 raft_consensus.cc:686] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [term 1 LEADER]: Becoming Leader. State: Replica: e7db430dacf04b929cc915527a3667f7, State: Running, Role: LEADER
I20260806 09:27:59.236033 16331 consensus_queue.cc:237] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } }
I20260806 09:27:59.246187 16333 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7db430dacf04b929cc915527a3667f7. Latest consensus state: current_term: 1 leader_uuid: "e7db430dacf04b929cc915527a3667f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } } }
I20260806 09:27:59.246821 16333 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:59.250623 16341 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:27:59.277871 16341 catalog_manager.cc:1484] Initializing Kudu cluster ID...
W20260806 09:27:59.297305 16112 tablet.cc:2369] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:27:59.301760 16280 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:59.263087 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47356 (local address 127.16.86.188:36289)
0806 09:27:59.263306 (+ 219us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.263313 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.275641 (+ 12328us) server_negotiation.cc:407] Connection header received
0806 09:27:59.275941 (+ 300us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.275955 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.276096 (+ 141us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.276200 (+ 104us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.278356 (+ 2156us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.279684 (+ 1328us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.282297 (+ 2613us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.283119 (+ 822us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.283226 (+ 107us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.290451 (+ 7225us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:59.290458 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:59.290464 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:59.290514 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:59.295046 (+ 4532us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.298779 (+ 3733us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.298786 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.298791 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.298959 (+ 168us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.299538 (+ 579us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.299545 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.299549 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.300012 (+ 463us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:59.300296 (+ 284us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:59.300950 (+ 654us) server_negotiation.cc:299] Negotiation successful
0806 09:27:59.301276 (+ 326us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20260806 09:27:59.324743 16281 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:59.257019 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38454 (local address 127.16.86.190:35111)
0806 09:27:59.257240 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.257248 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.262242 (+ 4994us) server_negotiation.cc:407] Connection header received
0806 09:27:59.264787 (+ 2545us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.264794 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.264904 (+ 110us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.264984 (+ 80us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.279066 (+ 14082us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.280238 (+ 1172us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.286195 (+ 5957us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.287052 (+ 857us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.287176 (+ 124us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.308900 (+ 21724us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:59.308909 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:59.308915 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:59.309014 (+ 99us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:59.313331 (+ 4317us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.320027 (+ 6696us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.320034 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.320038 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.320208 (+ 170us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.320692 (+ 484us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.320697 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.320701 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.321136 (+ 435us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:59.321430 (+ 294us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:59.324004 (+ 2574us) server_negotiation.cc:299] Negotiation successful
0806 09:27:59.324318 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20260806 09:27:59.326597 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:27:59.255459 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47016 (local address 127.16.86.189:33653)
0806 09:27:59.255693 (+ 234us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.255700 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.261687 (+ 5987us) server_negotiation.cc:407] Connection header received
0806 09:27:59.264278 (+ 2591us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.264285 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.264384 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.264470 (+ 86us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.282380 (+ 17910us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.283558 (+ 1178us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.288316 (+ 4758us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:27:59.289098 (+ 782us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:27:59.289203 (+ 105us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:27:59.314427 (+ 25224us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:27:59.314436 (+ 9us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:27:59.314441 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:27:59.314500 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:27:59.318934 (+ 4434us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.321993 (+ 3059us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.321999 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.322003 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.322180 (+ 177us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:27:59.322697 (+ 517us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:27:59.322702 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:27:59.322706 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:27:59.323148 (+ 442us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:27:59.323435 (+ 287us) server_negotiation.cc:1035] Waiting for connection context
0806 09:27:59.325823 (+ 2388us) server_negotiation.cc:299] Negotiation successful
0806 09:27:59.326076 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20260806 09:27:59.349696 16247 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [term 1 FOLLOWER]: Refusing update from remote peer e7db430dacf04b929cc915527a3667f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:27:59.354488 16333 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:27:59.362042 16144 raft_consensus.cc:1261] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [term 1 FOLLOWER]: Refusing update from remote peer e7db430dacf04b929cc915527a3667f7: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:27:59.366544 16333 consensus_queue.cc:1035] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:27:59.379668 16362 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260806 09:27:59.397264 16007 tablet.cc:2369] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:27:59.398973 16310 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7db430dacf04b929cc915527a3667f7. Latest consensus state: current_term: 1 leader_uuid: "e7db430dacf04b929cc915527a3667f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } } }
I20260806 09:27:59.399410 16310 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:27:59.407876 16361 mvcc.cc:205] Tried to move back new op lower bound from 7315490731364937728 to 7315490730964688896. Current Snapshot: MvccSnapshot[applied={T|T < 7315490731364937728}]
I20260806 09:27:59.408799 16353 mvcc.cc:205] Tried to move back new op lower bound from 7315490731364937728 to 7315490730964688896. Current Snapshot: MvccSnapshot[applied={T|T < 7315490731364937728}]
I20260806 09:27:59.434562 16310 sys_catalog.cc:455] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7db430dacf04b929cc915527a3667f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } } }
I20260806 09:27:59.434907 16310 sys_catalog.cc:458] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:27:59.436038 16331 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7db430dacf04b929cc915527a3667f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } } }
I20260806 09:27:59.436424 16331 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:59.442904 16341 catalog_manager.cc:1347] Generated new cluster ID: 24315a67164d4b7c86b3e0a4d0523b41
I20260806 09:27:59.443092 16341 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:27:59.446615 16286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e7db430dacf04b929cc915527a3667f7. Latest consensus state: current_term: 1 leader_uuid: "e7db430dacf04b929cc915527a3667f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } } }
I20260806 09:27:59.446964 16286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:27:59.459129 16333 sys_catalog.cc:455] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e7db430dacf04b929cc915527a3667f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } } }
I20260806 09:27:59.459723 16333 sys_catalog.cc:458] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7 [sys.catalog]: This master's current role is: LEADER
I20260806 09:27:59.461236 16286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e7db430dacf04b929cc915527a3667f7" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e7db430dacf04b929cc915527a3667f7" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 35111 } } peers { permanent_uuid: "af9e9c6ee7464f39bb0bea6437f1990f" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 33653 } } peers { permanent_uuid: "1567baa76dc2425ba644ab77a2afb919" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 36289 } } }
I20260806 09:27:59.461488 16286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:27:59.520041 16341 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:27:59.522375 16341 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:27:59.554883 16341 catalog_manager.cc:5868] T 00000000000000000000000000000000 P e7db430dacf04b929cc915527a3667f7: Generated new TSK 0
I20260806 09:27:59.556514 16341 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:27:59.716569 16281 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:27:59.695523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.16.86.190:35111)
0806 09:27:59.695830 (+ 307us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.695838 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.713894 (+ 18056us) server_negotiation.cc:407] Connection header received
0806 09:27:59.714507 (+ 613us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.714515 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.714639 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.714734 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.716208 (+ 1474us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38470: BlockingRecv error: recv got EOF from 127.0.0.1:38470 (error 108)
Metrics: {"server-negotiator.queue_time_us":98}
W20260806 09:27:59.719322 16284 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:27:59.697918 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47030 (local address 127.16.86.189:33653)
0806 09:27:59.698164 (+ 246us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.698172 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.717474 (+ 19302us) server_negotiation.cc:407] Connection header received
0806 09:27:59.717799 (+ 325us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.717806 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.717895 (+ 89us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.717988 (+ 93us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.719052 (+ 1064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47030: BlockingRecv error: recv got EOF from 127.0.0.1:47030 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
W20260806 09:27:59.723634 16280 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:27:59.720673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47362 (local address 127.16.86.188:36289)
0806 09:27:59.720857 (+ 184us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.720864 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.721741 (+ 877us) server_negotiation.cc:407] Connection header received
0806 09:27:59.722042 (+ 301us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.722049 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.722143 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.722239 (+ 96us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.723326 (+ 1087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47362: BlockingRecv error: recv got EOF from 127.0.0.1:47362 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
W20260806 09:27:59.746897 16284 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:27:59.743795 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47044 (local address 127.16.86.189:33653)
0806 09:27:59.744072 (+ 277us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.744080 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.744775 (+ 695us) server_negotiation.cc:407] Connection header received
0806 09:27:59.745108 (+ 333us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.745116 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.745303 (+ 187us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.745462 (+ 159us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.746549 (+ 1087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47044: BlockingRecv error: recv got EOF from 127.0.0.1:47044 (error 108)
Metrics: {"server-negotiator.queue_time_us":81}
W20260806 09:27:59.756366 16284 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:27:59.753526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47060 (local address 127.16.86.189:33653)
0806 09:27:59.753749 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.753757 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.754412 (+ 655us) server_negotiation.cc:407] Connection header received
0806 09:27:59.754664 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.754674 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.754804 (+ 130us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.754960 (+ 156us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.756077 (+ 1117us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47060: BlockingRecv error: recv got EOF from 127.0.0.1:47060 (error 108)
Metrics: {"server-negotiator.queue_time_us":91}
W20260806 09:27:59.765792 16284 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:27:59.762913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47076 (local address 127.16.86.189:33653)
0806 09:27:59.763198 (+ 285us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.763206 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.763936 (+ 730us) server_negotiation.cc:407] Connection header received
0806 09:27:59.764178 (+ 242us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.764189 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.764310 (+ 121us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.764445 (+ 135us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.765478 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47076: BlockingRecv error: recv got EOF from 127.0.0.1:47076 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
W20260806 09:27:59.776733 16284 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:27:59.773769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47082 (local address 127.16.86.189:33653)
0806 09:27:59.774070 (+ 301us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.774077 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.774757 (+ 680us) server_negotiation.cc:407] Connection header received
0806 09:27:59.775076 (+ 319us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.775084 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.775211 (+ 127us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.775363 (+ 152us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.776376 (+ 1013us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47082: BlockingRecv error: recv got EOF from 127.0.0.1:47082 (error 108)
Metrics: {"server-negotiator.queue_time_us":133}
W20260806 09:27:59.792153 16284 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:27:59.788948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47084 (local address 127.16.86.189:33653)
0806 09:27:59.789223 (+ 275us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.789230 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.789897 (+ 667us) server_negotiation.cc:407] Connection header received
0806 09:27:59.790183 (+ 286us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.790189 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.790301 (+ 112us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.790417 (+ 116us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.791801 (+ 1384us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47084: BlockingRecv error: recv got EOF from 127.0.0.1:47084 (error 108)
Metrics: {"server-negotiator.queue_time_us":96}
W20260806 09:27:59.816531 16284 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:27:59.813101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47098 (local address 127.16.86.189:33653)
0806 09:27:59.813368 (+ 267us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.813376 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.814324 (+ 948us) server_negotiation.cc:407] Connection header received
0806 09:27:59.814690 (+ 366us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.814697 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.814820 (+ 123us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.814965 (+ 145us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.816234 (+ 1269us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47098: BlockingRecv error: recv got EOF from 127.0.0.1:47098 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260806 09:27:59.860574 16284 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:27:59.857680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47106 (local address 127.16.86.189:33653)
0806 09:27:59.857946 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.857954 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.858700 (+ 746us) server_negotiation.cc:407] Connection header received
0806 09:27:59.858952 (+ 252us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.858958 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.859138 (+ 180us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.859266 (+ 128us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.860313 (+ 1047us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47106: BlockingRecv error: recv got EOF from 127.0.0.1:47106 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
W20260806 09:27:59.932133 16284 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:27:59.929316 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47112 (local address 127.16.86.189:33653)
0806 09:27:59.929555 (+ 239us) server_negotiation.cc:206] Beginning negotiation
0806 09:27:59.929561 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:27:59.930138 (+ 577us) server_negotiation.cc:407] Connection header received
0806 09:27:59.930444 (+ 306us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:27:59.930448 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:27:59.930544 (+ 96us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:27:59.930653 (+ 109us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:27:59.931883 (+ 1230us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47112: BlockingRecv error: recv got EOF from 127.0.0.1:47112 (error 108)
Metrics: {"server-negotiator.queue_time_us":101}
W20260806 09:28:00.068679 16284 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:00.066089 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47120 (local address 127.16.86.189:33653)
0806 09:28:00.066408 (+ 319us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:00.066415 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:00.066905 (+ 490us) server_negotiation.cc:407] Connection header received
0806 09:28:00.067164 (+ 259us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:00.067170 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:00.067269 (+ 99us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:00.067416 (+ 147us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:00.068349 (+ 933us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47120: BlockingRecv error: recv got EOF from 127.0.0.1:47120 (error 108)
Metrics: {"server-negotiator.queue_time_us":108}
I20260806 09:28:00.082118 16303 catalog_manager.cc:1259] Loaded cluster ID: 24315a67164d4b7c86b3e0a4d0523b41
I20260806 09:28:00.082266 16303 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919: loading cluster ID for follower catalog manager: success
I20260806 09:28:00.085438 16303 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919: acquiring CA information for follower catalog manager: success
I20260806 09:28:00.087872 16303 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 1567baa76dc2425ba644ab77a2afb919: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:28:00.178237 16326 catalog_manager.cc:1259] Loaded cluster ID: 24315a67164d4b7c86b3e0a4d0523b41
I20260806 09:28:00.178380 16326 catalog_manager.cc:1552] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f: loading cluster ID for follower catalog manager: success
I20260806 09:28:00.181358 16326 catalog_manager.cc:1574] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f: acquiring CA information for follower catalog manager: success
I20260806 09:28:00.183707 16326 catalog_manager.cc:1602] T 00000000000000000000000000000000 P af9e9c6ee7464f39bb0bea6437f1990f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:28:00.336658 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:00.329366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47132 (local address 127.16.86.189:33653)
0806 09:28:00.329665 (+ 299us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:00.329673 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:00.329843 (+ 170us) server_negotiation.cc:407] Connection header received
0806 09:28:00.330037 (+ 194us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:00.330043 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:00.330327 (+ 284us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:00.330523 (+ 196us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:28:00.331165 (+ 642us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:00.332060 (+ 895us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:00.333770 (+ 1710us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:00.334273 (+ 503us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:00.334430 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:00.335135 (+ 705us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:28:00.335903 (+ 768us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:28:00.336042 (+ 139us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:00.336251 (+ 209us) server_negotiation.cc:299] Negotiation successful
0806 09:28:00.336369 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":135}
I20260806 09:28:00.352699 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:00.345660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47364 (local address 127.16.86.188:36289)
0806 09:28:00.346369 (+ 709us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:00.346375 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:00.346399 (+ 24us) server_negotiation.cc:407] Connection header received
0806 09:28:00.346541 (+ 142us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:00.346546 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:00.346766 (+ 220us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:00.346984 (+ 218us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:28:00.347564 (+ 580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:00.348474 (+ 910us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:00.350054 (+ 1580us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:00.350547 (+ 493us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:00.350669 (+ 122us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:00.351280 (+ 611us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:28:00.352020 (+ 740us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:28:00.352153 (+ 133us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:00.352277 (+ 124us) server_negotiation.cc:299] Negotiation successful
0806 09:28:00.352368 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":559,"thread_start_us":426,"threads_started":1}
I20260806 09:28:00.370867 16474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:00.360769 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.16.86.190:35111)
0806 09:28:00.362397 (+ 1628us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:00.362406 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:00.362431 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:28:00.362603 (+ 172us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:00.362613 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:00.362954 (+ 341us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:00.363237 (+ 283us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:28:00.364418 (+ 1181us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:00.365455 (+ 1037us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:00.367720 (+ 2265us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:00.368217 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:00.368393 (+ 176us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:00.369209 (+ 816us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:28:00.369652 (+ 443us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:28:00.369808 (+ 156us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:00.370391 (+ 583us) server_negotiation.cc:299] Negotiation successful
0806 09:28:00.370499 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1472,"thread_start_us":1331,"threads_started":1}
I20260806 09:28:00.371587 16284 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:00.360823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47146 (local address 127.16.86.189:33653)
0806 09:28:00.361096 (+ 273us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:00.361106 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:00.364185 (+ 3079us) server_negotiation.cc:407] Connection header received
0806 09:28:00.364559 (+ 374us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:00.364567 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:00.364691 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:00.364805 (+ 114us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:28:00.365686 (+ 881us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:00.366504 (+ 818us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:00.368942 (+ 2438us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:00.369746 (+ 804us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:00.369903 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:00.370424 (+ 521us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:28:00.370563 (+ 139us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:28:00.370630 (+ 67us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:00.370914 (+ 284us) server_negotiation.cc:299] Negotiation successful
0806 09:28:00.371239 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20260806 09:28:00.371583 16463 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:00.362045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47378 (local address 127.16.86.188:36289)
0806 09:28:00.362522 (+ 477us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:00.362534 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:00.364493 (+ 1959us) server_negotiation.cc:407] Connection header received
0806 09:28:00.364669 (+ 176us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:00.364673 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:00.364756 (+ 83us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:00.364875 (+ 119us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:28:00.365728 (+ 853us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:00.366910 (+ 1182us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:00.369158 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:00.369944 (+ 786us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:00.370101 (+ 157us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:00.370629 (+ 528us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:28:00.370765 (+ 136us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:28:00.370877 (+ 112us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:00.371111 (+ 234us) server_negotiation.cc:299] Negotiation successful
0806 09:28:00.371223 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":212}
I20260806 09:28:00.384022 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 15979
I20260806 09:28:00.385444 16063 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:28:00.499907 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 15979
I20260806 09:28:00.530561 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 16082
W20260806 09:28:00.530942 16104 connection.cc:535] server connection from 127.0.0.1:34648 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260806 09:28:00.531958 16170 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:28:00.645948 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 16082
I20260806 09:28:00.683056 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 16192
I20260806 09:28:00.684607 16270 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:28:00.798041 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 16192
2026-08-06T09:28:00Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5439 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Aug 06 09:28:00 dist-test-slave-h3ft krb5kdc[16488](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Aug 06 09:28:00 dist-test-slave-h3ft krb5kdc[16488](info): set up 2 sockets
Aug 06 09:28:00 dist-test-slave-h3ft krb5kdc[16488](info): commencing operation
krb5kdc: starting...
W20260806 09:28:02.208493 16730 mini_kdc.cc:121] Time spent starting KDC: real 1.356s user 0.000s sys 0.058s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/test-user.keytab.
Aug 06 09:28:02 dist-test-slave-h3ft krb5kdc[16488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-08-06T09:28:02Z chronyd version 3.5 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -SECHASH -IPV6 +DEBUG)
2026-08-06T09:28:02Z Disabled control of system clock
WARNING: no policy specified for kudu/127.16.86.190@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.190@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
Entry for principal kudu/127.16.86.190 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab.
I20260806 09:28:02.656384 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:46357
--webserver_interface=127.16.86.190
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:46357,127.16.86.189:46825,127.16.86.188:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260806 09:28:03.150171 16512 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:28:03.150636 16512 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:28:03.150784 16512 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:28:03.150939 16512 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:28:03.159571 16512 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:28:03.159726 16512 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:28:03.159792 16512 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:28:03.159895 16512 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:28:03.160002 16512 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:28:03.172210 16512 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:46357,127.16.86.189:46825,127.16.86.188:44201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.190.keytab
--principal=kudu/127.16.86.190
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.190:46357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.16.86.190
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:28:03.174146 16512 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:28:03.176482 16512 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:28:03.192252 16525 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:28:03.192327 16521 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:28:03.192274 16518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:28:03.193902 16512 server_base.cc:1034] running on GCE node
I20260806 09:28:03.195106 16512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:28:03.197510 16512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:28:03.198829 16512 hybrid_clock.cc:648] HybridClock initialized: now 1786008483198740 us; error 53 us; skew 500 ppm
I20260806 09:28:03.199419 16512 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:28:03 dist-test-slave-h3ft krb5kdc[16488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008483, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:28:03.203630 16512 init.cc:377] Logged in from keytab as kudu/127.16.86.190@KRBTEST.COM (short username kudu)
I20260806 09:28:03.205713 16512 webserver.cc:460] Webserver started at http://127.16.86.190:34017/ using document root <none> and password file <none>
I20260806 09:28:03.206653 16512 fs_manager.cc:359] Metadata directory not provided
I20260806 09:28:03.206763 16512 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:28:03.207203 16512 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:28:03.210917 16512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/data/instance:
uuid: "d0683645ca7c41758405ff21832a4f25"
format_stamp: "Formatted at 2026-08-06 09:28:03 on dist-test-slave-h3ft"
server_key: "d15acd5a92d4bf51361133eeb4c30f5d"
server_key_iv: "cc5adc5440080a9e05e08c65f1ab4c81"
server_key_version: "encryptionkey@0"
I20260806 09:28:03.211961 16512 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/wal/instance:
uuid: "d0683645ca7c41758405ff21832a4f25"
format_stamp: "Formatted at 2026-08-06 09:28:03 on dist-test-slave-h3ft"
server_key: "d15acd5a92d4bf51361133eeb4c30f5d"
server_key_iv: "cc5adc5440080a9e05e08c65f1ab4c81"
server_key_version: "encryptionkey@0"
I20260806 09:28:03.219161 16512 fs_manager.cc:691] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260806 09:28:03.223829 16534 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:28:03.225790 16512 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260806 09:28:03.226051 16512 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/wal
uuid: "d0683645ca7c41758405ff21832a4f25"
format_stamp: "Formatted at 2026-08-06 09:28:03 on dist-test-slave-h3ft"
server_key: "d15acd5a92d4bf51361133eeb4c30f5d"
server_key_iv: "cc5adc5440080a9e05e08c65f1ab4c81"
server_key_version: "encryptionkey@0"
I20260806 09:28:03.226325 16512 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:28:03.245365 16512 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:28:03.251309 16512 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:28:03.251715 16512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:28:03.277942 16512 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.190:46357
I20260806 09:28:03.277994 16602 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.190:46357 every 8 connection(s)
I20260806 09:28:03.280011 16512 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/data/info.pb
I20260806 09:28:03.284859 16603 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:28:03.286401 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 16512
I20260806 09:28:03.286626 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-0/wal/instance
I20260806 09:28:03.287341 16730 external_mini_cluster.cc:1428] Setting key fb70e770b8fe957b1c3b19c49ee92577
I20260806 09:28:03.295996 16603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:28:03 dist-test-slave-h3ft krb5kdc[16488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008483, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:28:03.331247 16607 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.301393 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34038 (local address 127.16.86.190:46357)
0806 09:28:03.303985 (+ 2592us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:03.303992 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:03.304056 (+ 64us) server_negotiation.cc:407] Connection header received
0806 09:28:03.306920 (+ 2864us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:03.306933 (+ 13us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:03.307416 (+ 483us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:03.307729 (+ 313us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:03.309343 (+ 1614us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:03.311557 (+ 2214us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:03.313304 (+ 1747us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:03.314421 (+ 1117us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:03.314670 (+ 249us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:03.319784 (+ 5114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:03.319800 (+ 16us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:03.319827 (+ 27us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:03.319885 (+ 58us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:03.324620 (+ 4735us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:03.325672 (+ 1052us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:03.325685 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:03.325691 (+ 6us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:03.325918 (+ 227us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:03.326483 (+ 565us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:03.326491 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:03.326505 (+ 14us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:03.327516 (+ 1011us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:03.327758 (+ 242us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:03.328670 (+ 912us) server_negotiation.cc:299] Negotiation successful
0806 09:28:03.329205 (+ 535us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":46,"server-negotiator.queue_time_us":2344,"spinlock_wait_cycles":93568,"thread_start_us":2027,"threads_started":1}
I20260806 09:28:03.331408 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.301393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:46357 (local address 127.0.0.1:34038)
0806 09:28:03.302650 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:03.302719 (+ 69us) client_negotiation.cc:174] Beginning negotiation
0806 09:28:03.306295 (+ 3576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:28:03.307840 (+ 1545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:28:03.307846 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:28:03.308514 (+ 668us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:28:03.309067 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:03.309089 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:03.311797 (+ 2708us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:03.311803 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:03.313112 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:03.313127 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:03.314784 (+ 1657us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:03.314791 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:03.314956 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:03.315833 (+ 877us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:28:03.315859 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:28:03.319428 (+ 3569us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:28:03.324876 (+ 5448us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:03.324892 (+ 16us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:03.324904 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:03.325483 (+ 579us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:03.326103 (+ 620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:03.326106 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:03.326109 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:03.326340 (+ 231us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:03.327762 (+ 1422us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:28:03.327770 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:28:03.328048 (+ 278us) client_negotiation.cc:769] Sending connection context
0806 09:28:03.328489 (+ 441us) client_negotiation.cc:240] Negotiation successful
0806 09:28:03.329337 (+ 848us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":849,"mutex_wait_us":267,"thread_start_us":511,"threads_started":1}
WARNING: no policy specified for kudu/127.16.86.189@KRBTEST.COM; defaulting to no policy
I20260806 09:28:03.335417 16603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:28:03.336994 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.336436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38386)
0806 09:28:03.336710 (+ 274us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:03.336864 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":138}
W20260806 09:28:03.337368 16539 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.189@KRBTEST.COM" created.
I20260806 09:28:03.339521 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.339103 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38392)
0806 09:28:03.339295 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:03.339377 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":77}
W20260806 09:28:03.339877 16603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:46825: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.189 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.189.keytab.
Entry for principal kudu/127.16.86.189 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.189.keytab.
I20260806 09:28:03.388471 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.189:46825
--webserver_interface=127.16.86.189
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.189.keytab
--principal=kudu/127.16.86.189
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:46357,127.16.86.189:46825,127.16.86.188:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:28:03.389410 16603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } attempt: 1
I20260806 09:28:03.390959 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.390417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38400)
0806 09:28:03.390684 (+ 267us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:03.390801 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":85}
I20260806 09:28:03.392827 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.392109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38416)
0806 09:28:03.392380 (+ 271us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:03.392528 (+ 148us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
W20260806 09:28:03.393162 16603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:46825: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
I20260806 09:28:03.461472 16603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } attempt: 2
I20260806 09:28:03.463512 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.462610 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38424)
0806 09:28:03.463171 (+ 561us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:03.463311 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20260806 09:28:03.464852 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.464409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38426)
0806 09:28:03.464570 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:03.464685 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 09:28:03.465144 16603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:46825: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
I20260806 09:28:03.556564 16603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } attempt: 3
I20260806 09:28:03.558084 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.557549 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38438)
0806 09:28:03.557802 (+ 253us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:03.557915 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20260806 09:28:03.559305 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.558892 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38454)
0806 09:28:03.559084 (+ 192us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:03.559157 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20260806 09:28:03.559573 16603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:46825: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
I20260806 09:28:03.702889 16603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } attempt: 4
I20260806 09:28:03.704573 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.703894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38464)
0806 09:28:03.704289 (+ 395us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:03.704392 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":204}
I20260806 09:28:03.706064 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.705665 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38468)
0806 09:28:03.705826 (+ 161us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:03.705921 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20260806 09:28:03.706393 16603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.189:46825: Network error: Client connection negotiation failed: client connection to 127.16.86.189:46825: connect: Connection refused (error 111)
W20260806 09:28:03.822077 16614 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:28:03.822463 16614 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:28:03.822606 16614 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:28:03.822723 16614 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:28:03.830974 16614 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:28:03.831149 16614 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:28:03.831259 16614 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:28:03.831354 16614 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:28:03.831476 16614 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:28:03.844780 16614 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:46357,127.16.86.189:46825,127.16.86.188:44201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.189.keytab
--principal=kudu/127.16.86.189
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.189:46825
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.16.86.189
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:28:03.846329 16614 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:28:03.848538 16614 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:28:03.865578 16623 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:28:03.866037 16626 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:28:03.866542 16624 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:28:03.868059 16614 server_base.cc:1034] running on GCE node
I20260806 09:28:03.869405 16614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:28:03.871223 16614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:28:03.872536 16614 hybrid_clock.cc:648] HybridClock initialized: now 1786008483872423 us; error 80 us; skew 500 ppm
I20260806 09:28:03.873010 16614 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:28:03 dist-test-slave-h3ft krb5kdc[16488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008483, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.189@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:28:03.877204 16614 init.cc:377] Logged in from keytab as kudu/127.16.86.189@KRBTEST.COM (short username kudu)
I20260806 09:28:03.879256 16614 webserver.cc:460] Webserver started at http://127.16.86.189:32995/ using document root <none> and password file <none>
I20260806 09:28:03.880055 16614 fs_manager.cc:359] Metadata directory not provided
I20260806 09:28:03.880206 16614 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:28:03.880738 16614 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:28:03.884171 16614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/data/instance:
uuid: "0d95fd205b684662b5d51ad3dc927d83"
format_stamp: "Formatted at 2026-08-06 09:28:03 on dist-test-slave-h3ft"
server_key: "f23f4bb60aa686906b84c94ee4e4ef8a"
server_key_iv: "41aa23fd3dfaafda61aa1c89d52f7658"
server_key_version: "encryptionkey@0"
I20260806 09:28:03.885061 16614 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/wal/instance:
uuid: "0d95fd205b684662b5d51ad3dc927d83"
format_stamp: "Formatted at 2026-08-06 09:28:03 on dist-test-slave-h3ft"
server_key: "f23f4bb60aa686906b84c94ee4e4ef8a"
server_key_iv: "41aa23fd3dfaafda61aa1c89d52f7658"
server_key_version: "encryptionkey@0"
I20260806 09:28:03.891544 16614 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260806 09:28:03.896482 16633 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:28:03.898581 16614 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.004s sys 0.001s
I20260806 09:28:03.899112 16614 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/wal
uuid: "0d95fd205b684662b5d51ad3dc927d83"
format_stamp: "Formatted at 2026-08-06 09:28:03 on dist-test-slave-h3ft"
server_key: "f23f4bb60aa686906b84c94ee4e4ef8a"
server_key_iv: "41aa23fd3dfaafda61aa1c89d52f7658"
server_key_version: "encryptionkey@0"
I20260806 09:28:03.899399 16614 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:28:03.921420 16614 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:28:03.926991 16614 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:28:03.927356 16614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:28:03.952286 16614 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.189:46825
I20260806 09:28:03.952414 16707 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.189:46825 every 8 connection(s)
I20260806 09:28:03.953914 16614 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/data/info.pb
I20260806 09:28:03.959270 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 16614
I20260806 09:28:03.959403 16708 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:28:03.959542 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-1/wal/instance
I20260806 09:28:03.960291 16730 external_mini_cluster.cc:1428] Setting key d815619c208cacba41aee364cecec5a0
I20260806 09:28:03.971984 16708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:28:03 dist-test-slave-h3ft krb5kdc[16488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008483, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.189@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:28:04.001689 16713 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.977033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34048 (local address 127.16.86.190:46357)
0806 09:28:03.977773 (+ 740us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:03.977783 (+ 10us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:03.978931 (+ 1148us) server_negotiation.cc:407] Connection header received
0806 09:28:03.981213 (+ 2282us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:03.981225 (+ 12us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:03.981359 (+ 134us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:03.981558 (+ 199us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:03.983484 (+ 1926us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:03.984369 (+ 885us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:03.987551 (+ 3182us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:03.988127 (+ 576us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:03.988283 (+ 156us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:03.993911 (+ 5628us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:03.993918 (+ 7us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:03.993922 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:03.993988 (+ 66us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:03.997318 (+ 3330us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:03.998515 (+ 1197us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:03.998521 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:03.998524 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:03.998671 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:03.999264 (+ 593us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:03.999268 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:03.999270 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:03.999628 (+ 358us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:03.999810 (+ 182us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.000887 (+ 1077us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.001204 (+ 317us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":547,"thread_start_us":362,"threads_started":1}
I20260806 09:28:04.003312 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:03.977751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:46357 (local address 127.0.0.1:34048)
0806 09:28:03.978682 (+ 931us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:03.978752 (+ 70us) client_negotiation.cc:174] Beginning negotiation
0806 09:28:03.980839 (+ 2087us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:28:03.981835 (+ 996us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:28:03.981844 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:28:03.982438 (+ 594us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:28:03.983164 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:03.983184 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:03.984619 (+ 1435us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:03.984628 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:03.987300 (+ 2672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:03.987323 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:03.988443 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:03.988452 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:03.988858 (+ 406us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:03.989858 (+ 1000us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:28:03.989894 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:28:03.993457 (+ 3563us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:28:03.997608 (+ 4151us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:03.997626 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:03.997648 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:03.998325 (+ 677us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:03.998815 (+ 490us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:03.998820 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:03.998824 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:03.999126 (+ 302us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:03.999777 (+ 651us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:28:03.999787 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:28:04.000428 (+ 641us) client_negotiation.cc:769] Sending connection context
0806 09:28:04.000767 (+ 339us) client_negotiation.cc:240] Negotiation successful
0806 09:28:04.001510 (+ 743us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":622,"thread_start_us":374,"threads_started":1}
WARNING: no policy specified for kudu/127.16.86.188@KRBTEST.COM; defaulting to no policy
I20260806 09:28:04.005774 16603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } attempt: 5
I20260806 09:28:04.006153 16708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.16.86.188@KRBTEST.COM" created.
Aug 06 09:28:04 dist-test-slave-h3ft krb5kdc[16488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008483, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.189@KRBTEST.COM for kudu/127.16.86.189@KRBTEST.COM
Aug 06 09:28:04 dist-test-slave-h3ft krb5kdc[16488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008483, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for kudu/127.16.86.189@KRBTEST.COM
I20260806 09:28:04.027088 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.007368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38474)
0806 09:28:04.007771 (+ 403us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.007791 (+ 20us) client_negotiation.cc:174] Beginning negotiation
0806 09:28:04.007963 (+ 172us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:28:04.009863 (+ 1900us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:28:04.009868 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:28:04.010675 (+ 807us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:28:04.010932 (+ 257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.010948 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.013469 (+ 2521us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.013478 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.014710 (+ 1232us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.014724 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.015894 (+ 1170us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.015899 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.016154 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.016889 (+ 735us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:28:04.016917 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:28:04.019561 (+ 2644us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:28:04.024091 (+ 4530us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.024097 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.024101 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.024615 (+ 514us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.025068 (+ 453us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.025074 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.025078 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.025281 (+ 203us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.025975 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:28:04.025979 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:28:04.026156 (+ 177us) client_negotiation.cc:769] Sending connection context
0806 09:28:04.026311 (+ 155us) client_negotiation.cc:240] Negotiation successful
0806 09:28:04.026623 (+ 312us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":230,"mutex_wait_us":5}
I20260806 09:28:04.027160 16722 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.008463 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38474 (local address 127.16.86.189:46825)
0806 09:28:04.009453 (+ 990us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.009470 (+ 17us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.009501 (+ 31us) server_negotiation.cc:407] Connection header received
0806 09:28:04.009562 (+ 61us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.009570 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.009658 (+ 88us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.011171 (+ 1513us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.011591 (+ 420us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.013265 (+ 1674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.014877 (+ 1612us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.015723 (+ 846us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.015881 (+ 158us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.019872 (+ 3991us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.019893 (+ 21us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.019915 (+ 22us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.019956 (+ 41us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.023930 (+ 3974us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.024792 (+ 862us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.024799 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.024802 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.024949 (+ 147us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.025411 (+ 462us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.025415 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.025417 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.025836 (+ 419us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.026053 (+ 217us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.026468 (+ 415us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.026747 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":99,"server-negotiator.queue_time_us":808,"thread_start_us":507,"threads_started":1}
I20260806 09:28:04.030097 16606 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.006783 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38470)
0806 09:28:04.007137 (+ 354us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.007167 (+ 30us) client_negotiation.cc:174] Beginning negotiation
0806 09:28:04.007342 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:28:04.009734 (+ 2392us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:28:04.009741 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:28:04.010643 (+ 902us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:28:04.010876 (+ 233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.010889 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.012964 (+ 2075us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.012971 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.014277 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.014293 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.016022 (+ 1729us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.016046 (+ 24us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.016392 (+ 346us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.017168 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:28:04.017201 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:28:04.020485 (+ 3284us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:28:04.026184 (+ 5699us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.026193 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.026200 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.026898 (+ 698us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.028167 (+ 1269us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.028174 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.028178 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.028412 (+ 234us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.029026 (+ 614us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:28:04.029032 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:28:04.029206 (+ 174us) client_negotiation.cc:769] Sending connection context
0806 09:28:04.029360 (+ 154us) client_negotiation.cc:240] Negotiation successful
0806 09:28:04.029721 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":104}
I20260806 09:28:04.030169 16719 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.007213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38470 (local address 127.16.86.189:46825)
0806 09:28:04.008733 (+ 1520us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.008746 (+ 13us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.008791 (+ 45us) server_negotiation.cc:407] Connection header received
0806 09:28:04.008909 (+ 118us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.008924 (+ 15us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.009432 (+ 508us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.009713 (+ 281us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.011166 (+ 1453us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.012756 (+ 1590us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.014489 (+ 1733us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.015760 (+ 1271us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.016519 (+ 759us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.020866 (+ 4347us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.020878 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.020886 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.020948 (+ 62us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.025871 (+ 4923us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.027151 (+ 1280us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.027160 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.027753 (+ 593us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.027969 (+ 216us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.028571 (+ 602us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.028577 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.028580 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.028876 (+ 296us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.029053 (+ 177us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.029473 (+ 420us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.029697 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":64,"server-negotiator.queue_time_us":1237,"thread_start_us":1012,"threads_started":1}
I20260806 09:28:04.030763 16708 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:28:04.031543 16603 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:28:04.032581 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.031706 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59688)
0806 09:28:04.032099 (+ 393us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.032295 (+ 196us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
W20260806 09:28:04.032943 16639 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
I20260806 09:28:04.033392 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.032189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59696)
0806 09:28:04.032835 (+ 646us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.033068 (+ 233us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":345,"threads_started":1}
I20260806 09:28:04.034844 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.034293 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59712)
0806 09:28:04.034559 (+ 266us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.034666 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
W20260806 09:28:04.035146 16603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:44201: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
I20260806 09:28:04.045015 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.044170 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59714)
0806 09:28:04.044571 (+ 401us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.044749 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
W20260806 09:28:04.045485 16708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:44201: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.16.86.188 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.188.keytab.
Entry for principal kudu/127.16.86.188 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.188.keytab.
W20260806 09:28:04.067207 16730 mini_kdc.cc:262] Time spent creating service keytab for kudu/127.16.86.188: real 0.106s user 0.003s sys 0.043s
I20260806 09:28:04.068186 16730 external_mini_cluster.cc:1326] Running /tmp/dist-test-taskKAHevx/build/asan/bin/kudu
/tmp/dist-test-taskKAHevx/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.188:44201
--webserver_interface=127.16.86.188
--webserver_port=0
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.188.keytab
--principal=kudu/127.16.86.188
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--builtin_ntp_servers=127.16.86.148:41847
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.16.86.190:46357,127.16.86.189:46825,127.16.86.188:44201
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260806 09:28:04.086470 16603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } attempt: 1
I20260806 09:28:04.088363 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.087828 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59722)
0806 09:28:04.088074 (+ 246us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.088191 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":87}
I20260806 09:28:04.089957 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.089505 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59736)
0806 09:28:04.089740 (+ 235us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.089831 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":84}
W20260806 09:28:04.090371 16603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:44201: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
I20260806 09:28:04.094941 16708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } attempt: 1
I20260806 09:28:04.096537 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.095960 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59746)
0806 09:28:04.096239 (+ 279us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.096360 (+ 121us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":122}
I20260806 09:28:04.097640 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.097278 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59748)
0806 09:28:04.097446 (+ 168us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.097530 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20260806 09:28:04.097854 16708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:44201: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
I20260806 09:28:04.158670 16603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } attempt: 2
I20260806 09:28:04.160621 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.159877 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59750)
0806 09:28:04.160254 (+ 377us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.160403 (+ 149us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":166}
I20260806 09:28:04.161742 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.161367 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59754)
0806 09:28:04.161541 (+ 174us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.161625 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
W20260806 09:28:04.161962 16603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:44201: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
I20260806 09:28:04.166070 16708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } attempt: 2
I20260806 09:28:04.167512 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.166978 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59760)
0806 09:28:04.167246 (+ 268us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.167354 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":126}
I20260806 09:28:04.168606 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.168248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59762)
0806 09:28:04.168410 (+ 162us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.168488 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20260806 09:28:04.168808 16708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:44201: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
I20260806 09:28:04.260052 16708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } attempt: 3
I20260806 09:28:04.261698 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.261030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59772)
0806 09:28:04.261324 (+ 294us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.261476 (+ 152us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
I20260806 09:28:04.262945 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.262487 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59784)
0806 09:28:04.262746 (+ 259us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.262833 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
W20260806 09:28:04.263304 16708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:44201: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
I20260806 09:28:04.268215 16603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } attempt: 3
I20260806 09:28:04.269490 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.269021 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59788)
0806 09:28:04.269226 (+ 205us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.269324 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":70}
I20260806 09:28:04.271080 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.270570 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59794)
0806 09:28:04.270802 (+ 232us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.270894 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
W20260806 09:28:04.271382 16603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:44201: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
I20260806 09:28:04.406654 16708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } attempt: 4
I20260806 09:28:04.408144 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.407631 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59798)
0806 09:28:04.407845 (+ 214us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.407971 (+ 126us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 09:28:04.409238 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.408897 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59808)
0806 09:28:04.409054 (+ 157us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.409123 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20260806 09:28:04.409461 16708 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:44201: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
I20260806 09:28:04.448712 16603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } attempt: 4
I20260806 09:28:04.450042 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.449547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59818)
0806 09:28:04.449783 (+ 236us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.449882 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20260806 09:28:04.451151 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.450716 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59832)
0806 09:28:04.450899 (+ 183us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.450972 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20260806 09:28:04.451366 16603 consensus_peers.cc:646] Error getting permanent uuid from config peer 127.16.86.188:44201: Network error: Client connection negotiation failed: client connection to 127.16.86.188:44201: connect: Connection refused (error 111)
W20260806 09:28:04.526719 16728 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20260806 09:28:04.527280 16728 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20260806 09:28:04.527417 16728 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260806 09:28:04.527551 16728 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20260806 09:28:04.536767 16728 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20260806 09:28:04.536916 16728 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20260806 09:28:04.537025 16728 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20260806 09:28:04.537108 16728 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260806 09:28:04.537209 16728 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20260806 09:28:04.549292 16728 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.16.86.148:41847
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.16.86.190:46357,127.16.86.189:46825,127.16.86.188:44201
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/krb5kdc/kudu_127.16.86.188.keytab
--principal=kudu/127.16.86.188
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.16.86.188:44201
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.16.86.188
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.18.0-SNAPSHOT
revision c2d25d5f57d102d9ad9869eaf208d035a22e18aa
build type FASTDEBUG
built by systest at 06 Aug 2026 07:10:43 UTC on ccycloud.quasar-ubmxwc.root.comops.site
build id 4569745
ASAN enabled
I20260806 09:28:04.551286 16728 env_posix.cc:2256] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260806 09:28:04.553102 16728 file_cache.cc:493] Constructed file cache file cache with capacity 419430
W20260806 09:28:04.568472 16739 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:28:04.568593 16741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260806 09:28:04.568607 16743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260806 09:28:04.569870 16728 server_base.cc:1034] running on GCE node
I20260806 09:28:04.571144 16728 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260806 09:28:04.573269 16728 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260806 09:28:04.574565 16728 hybrid_clock.cc:648] HybridClock initialized: now 1786008484574460 us; error 69 us; skew 500 ppm
I20260806 09:28:04.575098 16728 server_base.cc:834] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Aug 06 09:28:04 dist-test-slave-h3ft krb5kdc[16488](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008484, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.188@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260806 09:28:04.579560 16728 init.cc:377] Logged in from keytab as kudu/127.16.86.188@KRBTEST.COM (short username kudu)
I20260806 09:28:04.581961 16728 webserver.cc:460] Webserver started at http://127.16.86.188:43305/ using document root <none> and password file <none>
I20260806 09:28:04.583320 16728 fs_manager.cc:359] Metadata directory not provided
I20260806 09:28:04.583508 16728 fs_manager.cc:365] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260806 09:28:04.583948 16728 server_base.cc:882] This appears to be a new deployment of Kudu; creating new FS layout
I20260806 09:28:04.587597 16728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/data/instance:
uuid: "582483e4cf4c4ce7a487b7c813931b08"
format_stamp: "Formatted at 2026-08-06 09:28:04 on dist-test-slave-h3ft"
server_key: "b869250a6febbd5cfde2486adbbcb594"
server_key_iv: "6e55c576e111d1074cae32c3b62cd72e"
server_key_version: "encryptionkey@0"
I20260806 09:28:04.588986 16728 fs_manager.cc:1063] Generated new instance metadata in path /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/wal/instance:
uuid: "582483e4cf4c4ce7a487b7c813931b08"
format_stamp: "Formatted at 2026-08-06 09:28:04 on dist-test-slave-h3ft"
server_key: "b869250a6febbd5cfde2486adbbcb594"
server_key_iv: "6e55c576e111d1074cae32c3b62cd72e"
server_key_version: "encryptionkey@0"
I20260806 09:28:04.595860 16728 fs_manager.cc:691] Time spent creating directory manager: real 0.006s user 0.004s sys 0.003s
I20260806 09:28:04.601141 16757 log_block_manager.cc:3774] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260806 09:28:04.603225 16728 fs_manager.cc:725] Time spent opening block manager: real 0.004s user 0.003s sys 0.003s
I20260806 09:28:04.603583 16728 fs_manager.cc:642] Opened local filesystem: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/data,/tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/wal
uuid: "582483e4cf4c4ce7a487b7c813931b08"
format_stamp: "Formatted at 2026-08-06 09:28:04 on dist-test-slave-h3ft"
server_key: "b869250a6febbd5cfde2486adbbcb594"
server_key_iv: "6e55c576e111d1074cae32c3b62cd72e"
server_key_version: "encryptionkey@0"
I20260806 09:28:04.603986 16728 fs_report.cc:383] FS layout report
--------------------
wal directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260806 09:28:04.629225 16728 rpc_server.cc:225] running with OpenSSL 1.0.2k-fips 26 Jan 2017
I20260806 09:28:04.635237 16728 env_posix.cc:2256] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260806 09:28:04.635494 16728 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260806 09:28:04.660743 16728 rpc_server.cc:307] RPC server started. Bound to: 127.16.86.188:44201
I20260806 09:28:04.660786 16826 acceptor_pool.cc:254] collecting diagnostics on the listening RPC socket 127.16.86.188:44201 every 8 connection(s)
I20260806 09:28:04.662992 16728 server_base.cc:1166] Dumped server information to /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/data/info.pb
I20260806 09:28:04.668663 16827 data_dirs.cc:395] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260806 09:28:04.669384 16730 external_mini_cluster.cc:1388] Started /tmp/dist-test-taskKAHevx/build/asan/bin/kudu as pid 16728
I20260806 09:28:04.669621 16730 external_mini_cluster.cc:1402] Reading /tmp/dist-test-taskKAHevx/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1786008061536593-16730-0/minicluster-data/master-2/wal/instance
I20260806 09:28:04.670171 16730 external_mini_cluster.cc:1428] Setting key 92430f2045c19776d7c86240f1969fbe
I20260806 09:28:04.682288 16827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } has no permanent_uuid. Determining permanent_uuid...
Aug 06 09:28:04 dist-test-slave-h3ft krb5kdc[16488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.188@KRBTEST.COM
Aug 06 09:28:04 dist-test-slave-h3ft krb5kdc[16488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008484, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.188@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:28:04.702097 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.672602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59836 (local address 127.16.86.188:44201)
0806 09:28:04.674060 (+ 1458us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.674074 (+ 14us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.674123 (+ 49us) server_negotiation.cc:407] Connection header received
0806 09:28:04.677242 (+ 3119us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.677281 (+ 39us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.677817 (+ 536us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.678234 (+ 417us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.680957 (+ 2723us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.682417 (+ 1460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.684769 (+ 2352us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.685571 (+ 802us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.685758 (+ 187us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.692438 (+ 6680us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.692448 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.692465 (+ 17us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.692535 (+ 70us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.696838 (+ 4303us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.697861 (+ 1023us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.697872 (+ 11us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.697885 (+ 13us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.698159 (+ 274us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.698695 (+ 536us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.698698 (+ 3us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.698700 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.699467 (+ 767us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.699741 (+ 274us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.700246 (+ 505us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.700791 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1013,"thread_start_us":673,"threads_started":1}
I20260806 09:28:04.705915 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.686760 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.16.86.190:46357)
0806 09:28:04.687589 (+ 829us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.687598 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.688339 (+ 741us) server_negotiation.cc:407] Connection header received
0806 09:28:04.688578 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.688586 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.688717 (+ 131us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.688921 (+ 204us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.690571 (+ 1650us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.691571 (+ 1000us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.693471 (+ 1900us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.694034 (+ 563us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.694205 (+ 171us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.698493 (+ 4288us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.698501 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.698506 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.698556 (+ 50us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.702531 (+ 3975us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.703532 (+ 1001us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.703538 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.703541 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.703672 (+ 131us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.704143 (+ 471us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.704147 (+ 4us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.704149 (+ 2us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.704457 (+ 308us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.704665 (+ 208us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.705248 (+ 583us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.705494 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":627,"thread_start_us":523,"threads_started":1}
I20260806 09:28:04.705968 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.687125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.190:46357 (local address 127.0.0.1:34058)
0806 09:28:04.688105 (+ 980us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.688156 (+ 51us) client_negotiation.cc:174] Beginning negotiation
0806 09:28:04.688388 (+ 232us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:28:04.689018 (+ 630us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:28:04.689024 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:28:04.689850 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:28:04.690314 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.690329 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.691775 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.691782 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.693234 (+ 1452us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.693252 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.694304 (+ 1052us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.694311 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.694506 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.695281 (+ 775us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:28:04.695302 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:28:04.698232 (+ 2930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:28:04.702713 (+ 4481us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.702731 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.702739 (+ 8us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.703401 (+ 662us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.703797 (+ 396us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.703804 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.703807 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.703988 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.704626 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:28:04.704637 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:28:04.704900 (+ 263us) client_negotiation.cc:769] Sending connection context
0806 09:28:04.705125 (+ 225us) client_negotiation.cc:240] Negotiation successful
0806 09:28:04.705498 (+ 373us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":673,"thread_start_us":430,"threads_started":1}
I20260806 09:28:04.708132 16827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:28:04.708690 16708 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } attempt: 5
Aug 06 09:28:04 dist-test-slave-h3ft krb5kdc[16488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008484, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.188@KRBTEST.COM for kudu/127.16.86.189@KRBTEST.COM
Aug 06 09:28:04 dist-test-slave-h3ft krb5kdc[16488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008483, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.189@KRBTEST.COM for kudu/127.16.86.188@KRBTEST.COM
Aug 06 09:28:04 dist-test-slave-h3ft krb5kdc[16488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.190@KRBTEST.COM
I20260806 09:28:04.726835 16714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.709538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59840)
0806 09:28:04.709715 (+ 177us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.709737 (+ 22us) client_negotiation.cc:174] Beginning negotiation
0806 09:28:04.709854 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:28:04.710308 (+ 454us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:28:04.710314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:28:04.710890 (+ 576us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:28:04.711187 (+ 297us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.711202 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.712657 (+ 1455us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.712663 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.714335 (+ 1672us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.714349 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.715208 (+ 859us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.715217 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.715484 (+ 267us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.716105 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:28:04.716134 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:28:04.719546 (+ 3412us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:28:04.723585 (+ 4039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.723591 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.723594 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.724111 (+ 517us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.724616 (+ 505us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.724628 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.724631 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.724768 (+ 137us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.725409 (+ 641us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:28:04.725415 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:28:04.725608 (+ 193us) client_negotiation.cc:769] Sending connection context
0806 09:28:04.726021 (+ 413us) client_negotiation.cc:240] Negotiation successful
0806 09:28:04.726386 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20260806 09:28:04.726891 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.709709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59840 (local address 127.16.86.188:44201)
0806 09:28:04.709902 (+ 193us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.709908 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.709928 (+ 20us) server_negotiation.cc:407] Connection header received
0806 09:28:04.709988 (+ 60us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.709995 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.710087 (+ 92us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.711362 (+ 1275us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.711658 (+ 296us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.712473 (+ 815us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.714508 (+ 2035us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.714968 (+ 460us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.715166 (+ 198us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.719821 (+ 4655us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.719829 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.719835 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.719884 (+ 49us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.723382 (+ 3498us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.724251 (+ 869us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.724260 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.724265 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.724422 (+ 157us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.724885 (+ 463us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.724891 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.724894 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.725288 (+ 394us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.725459 (+ 171us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.726150 (+ 691us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.726450 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":44,"server-negotiator.queue_time_us":69,"spinlock_wait_cycles":31616}
I20260806 09:28:04.727984 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.709430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.189:46825 (local address 127.0.0.1:38478)
0806 09:28:04.709867 (+ 437us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.709888 (+ 21us) client_negotiation.cc:174] Beginning negotiation
0806 09:28:04.709986 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:28:04.710808 (+ 822us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:28:04.710813 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:28:04.711375 (+ 562us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:28:04.711646 (+ 271us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.711657 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.713038 (+ 1381us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.713045 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.714151 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.714164 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.715386 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.715390 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.715557 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.716267 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:28:04.716287 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:28:04.719536 (+ 3249us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:28:04.724127 (+ 4591us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.724133 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.724139 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.724804 (+ 665us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.725321 (+ 517us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.725328 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.725335 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.725545 (+ 210us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.726307 (+ 762us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:28:04.726311 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:28:04.726465 (+ 154us) client_negotiation.cc:769] Sending connection context
0806 09:28:04.726598 (+ 133us) client_negotiation.cc:240] Negotiation successful
0806 09:28:04.727721 (+ 1123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":312,"mutex_wait_us":3}
I20260806 09:28:04.728652 16603 consensus_peers.cc:656] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } attempt: 5
I20260806 09:28:04.729187 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.706161 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34068 (local address 127.16.86.190:46357)
0806 09:28:04.706405 (+ 244us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.706411 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.706911 (+ 500us) server_negotiation.cc:407] Connection header received
0806 09:28:04.707248 (+ 337us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.707258 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.707374 (+ 116us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.707492 (+ 118us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.709421 (+ 1929us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.710874 (+ 1453us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.712484 (+ 1610us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.713158 (+ 674us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.713306 (+ 148us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.720420 (+ 7114us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.720433 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.720440 (+ 7us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.720499 (+ 59us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.724761 (+ 4262us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.725779 (+ 1018us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.725785 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.725788 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.726048 (+ 260us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.726558 (+ 510us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.726564 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.726568 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.727036 (+ 468us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.727325 (+ 289us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.728567 (+ 1242us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.728830 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":40,"server-negotiator.queue_time_us":118}
I20260806 09:28:04.729625 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.709882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38478 (local address 127.16.86.189:46825)
0806 09:28:04.710470 (+ 588us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.710476 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.710493 (+ 17us) server_negotiation.cc:407] Connection header received
0806 09:28:04.710546 (+ 53us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.710551 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.710624 (+ 73us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.710754 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.711838 (+ 1084us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.712887 (+ 1049us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.714345 (+ 1458us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.715151 (+ 806us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.715370 (+ 219us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.719849 (+ 4479us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.719855 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.719859 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.719892 (+ 33us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.723906 (+ 4014us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.724990 (+ 1084us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.724995 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.724998 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.725178 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.725668 (+ 490us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.725674 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.725679 (+ 5us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.726155 (+ 476us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.728463 (+ 2308us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.728663 (+ 200us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.728979 (+ 316us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":469,"spinlock_wait_cycles":92416,"thread_start_us":328,"threads_started":1}
I20260806 09:28:04.735301 16827 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } has no permanent_uuid. Determining permanent_uuid...
I20260806 09:28:04.735924 16708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83: Bootstrap starting.
I20260806 09:28:04.740589 16708 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83: Neither blocks nor log segments found. Creating new log.
I20260806 09:28:04.742149 16708 log.cc:824] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83: Log is configured to *not* fsync() on all Append() calls
Aug 06 09:28:04 dist-test-slave-h3ft krb5kdc[16488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008482, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.16.86.189@KRBTEST.COM
Aug 06 09:28:04 dist-test-slave-h3ft krb5kdc[16488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008483, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.190@KRBTEST.COM for kudu/127.16.86.188@KRBTEST.COM
I20260806 09:28:04.751082 16708 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83: No bootstrap required, opened a new log
Aug 06 09:28:04 dist-test-slave-h3ft krb5kdc[16488](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1786008484, etypes {rep=17 tkt=17 ses=17}, kudu/127.16.86.188@KRBTEST.COM for kudu/127.16.86.188@KRBTEST.COM
I20260806 09:28:04.756073 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.732029 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38492 (local address 127.16.86.189:46825)
0806 09:28:04.732228 (+ 199us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.732252 (+ 24us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.732277 (+ 25us) server_negotiation.cc:407] Connection header received
0806 09:28:04.732459 (+ 182us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.732463 (+ 4us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.732537 (+ 74us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.732648 (+ 111us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.735271 (+ 2623us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.736469 (+ 1198us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.738326 (+ 1857us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.739175 (+ 849us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.739329 (+ 154us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.746663 (+ 7334us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.746673 (+ 10us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.746678 (+ 5us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.746742 (+ 64us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.751540 (+ 4798us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.752722 (+ 1182us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.752730 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.752734 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.752906 (+ 172us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.753411 (+ 505us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.753420 (+ 9us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.753424 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.753881 (+ 457us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.754239 (+ 358us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.755195 (+ 956us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.755490 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20260806 09:28:04.762930 16831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.735701 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59856 (local address 127.16.86.188:44201)
0806 09:28:04.735924 (+ 223us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.735931 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.735953 (+ 22us) server_negotiation.cc:407] Connection header received
0806 09:28:04.736020 (+ 67us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.736027 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.736124 (+ 97us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.737641 (+ 1517us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.737856 (+ 215us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.739492 (+ 1636us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.742101 (+ 2609us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.743161 (+ 1060us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.743324 (+ 163us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.749591 (+ 6267us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.749603 (+ 12us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.749609 (+ 6us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.749666 (+ 57us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.754990 (+ 5324us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.756652 (+ 1662us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.756659 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.756663 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.756903 (+ 240us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.757514 (+ 611us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.757522 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.757526 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.758297 (+ 771us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.758478 (+ 181us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.762139 (+ 3661us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.762450 (+ 311us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20260806 09:28:04.760447 16726 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.729582 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59856)
0806 09:28:04.735247 (+ 5665us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.735271 (+ 24us) client_negotiation.cc:174] Beginning negotiation
0806 09:28:04.735420 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:28:04.736346 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:28:04.736353 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:28:04.737045 (+ 692us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:28:04.737397 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.737416 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.739750 (+ 2334us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.739760 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.741752 (+ 1992us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.741782 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.743381 (+ 1599us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.743388 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.743623 (+ 235us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.744728 (+ 1105us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:28:04.744767 (+ 39us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:28:04.749163 (+ 4396us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:28:04.755274 (+ 6111us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.755282 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.755287 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.756046 (+ 759us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.757085 (+ 1039us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.757090 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.757094 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.757344 (+ 250us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.758621 (+ 1277us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:28:04.758630 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:28:04.758975 (+ 345us) client_negotiation.cc:769] Sending connection context
0806 09:28:04.759198 (+ 223us) client_negotiation.cc:240] Negotiation successful
0806 09:28:04.760019 (+ 821us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5482}
I20260806 09:28:04.763621 16730 external_mini_cluster.cc:930] 0 TS(s) registered with all masters
I20260806 09:28:04.765323 16833 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.736308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.16.86.188:44201 (local address 127.0.0.1:59862)
0806 09:28:04.738803 (+ 2495us) negotiation.cc:107] Waiting for socket to connect
0806 09:28:04.738834 (+ 31us) client_negotiation.cc:174] Beginning negotiation
0806 09:28:04.738967 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
0806 09:28:04.744376 (+ 5409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
0806 09:28:04.744384 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
0806 09:28:04.744966 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
0806 09:28:04.745258 (+ 292us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.745273 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.747895 (+ 2622us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.747902 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.749434 (+ 1532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
0806 09:28:04.749454 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.750661 (+ 1207us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.750666 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
0806 09:28:04.750912 (+ 246us) client_negotiation.cc:548] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.751710 (+ 798us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
0806 09:28:04.751742 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
0806 09:28:04.755222 (+ 3480us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
0806 09:28:04.760445 (+ 5223us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.760451 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.760455 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.761323 (+ 868us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.761831 (+ 508us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
0806 09:28:04.761837 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
0806 09:28:04.761841 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
0806 09:28:04.762022 (+ 181us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
0806 09:28:04.762845 (+ 823us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
0806 09:28:04.762850 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
0806 09:28:04.764176 (+ 1326us) client_negotiation.cc:769] Sending connection context
0806 09:28:04.764395 (+ 219us) client_negotiation.cc:240] Negotiation successful
0806 09:28:04.764835 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2359}
I20260806 09:28:04.766903 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.736770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59862 (local address 127.16.86.188:44201)
0806 09:28:04.743684 (+ 6914us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.743693 (+ 9us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.743737 (+ 44us) server_negotiation.cc:407] Connection header received
0806 09:28:04.743976 (+ 239us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.744005 (+ 29us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.744149 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.745424 (+ 1275us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.745799 (+ 375us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.747729 (+ 1930us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.749654 (+ 1925us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.750474 (+ 820us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.752150 (+ 1676us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.755470 (+ 3320us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.755476 (+ 6us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.755480 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.755517 (+ 37us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.760236 (+ 4719us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.761496 (+ 1260us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.761502 (+ 6us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.761506 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.761686 (+ 180us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.762150 (+ 464us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.762155 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.762159 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.762585 (+ 426us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.762775 (+ 190us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.766158 (+ 3383us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.766447 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6535,"thread_start_us":6293,"threads_started":1}
I20260806 09:28:04.766901 16708 raft_consensus.cc:348] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } }
I20260806 09:28:04.767505 16708 raft_consensus.cc:729] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d95fd205b684662b5d51ad3dc927d83, State: Initialized, Role: FOLLOWER
I20260806 09:28:04.768227 16708 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } }
I20260806 09:28:04.769189 16603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25: Bootstrap starting.
I20260806 09:28:04.769520 16839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } } }
I20260806 09:28:04.770093 16839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:28:04.770951 16708 sys_catalog.cc:564] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:28:04.774127 16603 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25: Neither blocks nor log segments found. Creating new log.
I20260806 09:28:04.775693 16603 log.cc:824] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25: Log is configured to *not* fsync() on all Append() calls
I20260806 09:28:04.772069 16567 tablet_service.cc:1939] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d0683645ca7c41758405ff21832a4f25"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:34068
I20260806 09:28:04.780019 16827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08: Bootstrap starting.
I20260806 09:28:04.785785 16603 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25: No bootstrap required, opened a new log
I20260806 09:28:04.786401 16827 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08: Neither blocks nor log segments found. Creating new log.
I20260806 09:28:04.789470 16827 log.cc:824] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08: Log is configured to *not* fsync() on all Append() calls
I20260806 09:28:04.793943 16603 raft_consensus.cc:348] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } }
I20260806 09:28:04.794378 16603 raft_consensus.cc:729] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0683645ca7c41758405ff21832a4f25, State: Initialized, Role: FOLLOWER
I20260806 09:28:04.794637 16827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08: No bootstrap required, opened a new log
I20260806 09:28:04.794906 16603 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } }
I20260806 09:28:04.796586 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } } }
I20260806 09:28:04.797366 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:28:04.797747 16603 sys_catalog.cc:564] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:28:04.799187 16567 raft_consensus.cc:482] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20260806 09:28:04.799431 16567 raft_consensus.cc:3037] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:28:04.802462 16857 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:28:04.804008 16567 raft_consensus.cc:504] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } }
I20260806 09:28:04.806366 16567 leader_election.cc:288] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [CANDIDATE]: Term 1 election: Requested vote from peers 0d95fd205b684662b5d51ad3dc927d83 (127.16.86.189:46825), 582483e4cf4c4ce7a487b7c813931b08 (127.16.86.188:44201)
I20260806 09:28:04.808117 16675 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0683645ca7c41758405ff21832a4f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "0d95fd205b684662b5d51ad3dc927d83"
I20260806 09:28:04.808779 16675 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:28:04.813803 16675 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0683645ca7c41758405ff21832a4f25 in term 1.
I20260806 09:28:04.814442 16827 raft_consensus.cc:348] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } }
I20260806 09:28:04.814849 16539 leader_election.cc:302] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d95fd205b684662b5d51ad3dc927d83, d0683645ca7c41758405ff21832a4f25; no voters:
I20260806 09:28:04.815208 16827 raft_consensus.cc:729] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 582483e4cf4c4ce7a487b7c813931b08, State: Initialized, Role: FOLLOWER
I20260806 09:28:04.815615 16856 raft_consensus.cc:2781] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [term 1 FOLLOWER]: Leader election won for term 1
I20260806 09:28:04.816057 16827 consensus_queue.cc:260] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } }
I20260806 09:28:04.824134 16866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } } }
I20260806 09:28:04.824873 16866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:28:04.826134 16827 sys_catalog.cc:564] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [sys.catalog]: configured and running, proceeding with master startup.
I20260806 09:28:04.826242 16793 tablet_service.cc:1812] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0683645ca7c41758405ff21832a4f25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "582483e4cf4c4ce7a487b7c813931b08"
I20260806 09:28:04.806420 16614 master_runner.cc:186] Error getting master registration for 127.16.86.190:46357: OK, instance_id { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" instance_seqno: 1786008483270396 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:28:04.827044 16614 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 09:28:04.827078 16793 raft_consensus.cc:3037] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [term 0 FOLLOWER]: Advancing to term 1
I20260806 09:28:04.827311 16856 raft_consensus.cc:686] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [term 1 LEADER]: Becoming Leader. State: Replica: d0683645ca7c41758405ff21832a4f25, State: Running, Role: LEADER
I20260806 09:28:04.828235 16856 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } }
I20260806 09:28:04.832463 16793 raft_consensus.cc:2454] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0683645ca7c41758405ff21832a4f25 in term 1.
I20260806 09:28:04.840603 16864 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0683645ca7c41758405ff21832a4f25. Latest consensus state: current_term: 1 leader_uuid: "d0683645ca7c41758405ff21832a4f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } } }
I20260806 09:28:04.841455 16864 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [sys.catalog]: This master's current role is: LEADER
I20260806 09:28:04.851910 16878 catalog_manager.cc:1475] Loading table and tablet metadata into memory...
I20260806 09:28:04.862883 16878 catalog_manager.cc:1484] Initializing Kudu cluster ID...
I20260806 09:28:04.890002 16832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.855072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34080 (local address 127.16.86.190:46357)
0806 09:28:04.855279 (+ 207us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.855285 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.856632 (+ 1347us) server_negotiation.cc:407] Connection header received
0806 09:28:04.857430 (+ 798us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.857440 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.857753 (+ 313us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.857965 (+ 212us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.863201 (+ 5236us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.865344 (+ 2143us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.868601 (+ 3257us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.870609 (+ 2008us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.870813 (+ 204us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.875882 (+ 5069us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.875890 (+ 8us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.875894 (+ 4us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.875969 (+ 75us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.881836 (+ 5867us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.883524 (+ 1688us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.883536 (+ 12us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.883543 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.883825 (+ 282us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.884531 (+ 706us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.884541 (+ 10us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.884548 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.885233 (+ 685us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.885634 (+ 401us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.888727 (+ 3093us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.889200 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20260806 09:28:04.903885 16764 tablet.cc:2369] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:28:04.927927 16793 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [term 1 FOLLOWER]: Refusing update from remote peer d0683645ca7c41758405ff21832a4f25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:28:04.927989 16675 raft_consensus.cc:1261] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [term 1 FOLLOWER]: Refusing update from remote peer d0683645ca7c41758405ff21832a4f25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260806 09:28:04.929658 16864 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260806 09:28:04.931430 16641 tablet.cc:2369] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260806 09:28:04.933074 16856 consensus_queue.cc:1035] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260806 09:28:04.939764 16835 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.855986 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38498 (local address 127.16.86.189:46825)
0806 09:28:04.856207 (+ 221us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.856215 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.870991 (+ 14776us) server_negotiation.cc:407] Connection header received
0806 09:28:04.891494 (+ 20503us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.891508 (+ 14us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.891669 (+ 161us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.891783 (+ 114us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.898299 (+ 6516us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.899980 (+ 1681us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.910263 (+ 10283us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.911463 (+ 1200us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.911642 (+ 179us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.927792 (+ 16150us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.927831 (+ 39us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.927841 (+ 10us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.927935 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.934147 (+ 6212us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.935303 (+ 1156us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.935310 (+ 7us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.935314 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.935473 (+ 159us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.935959 (+ 486us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.935964 (+ 5us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.935968 (+ 4us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.936748 (+ 780us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.937040 (+ 292us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.938797 (+ 1757us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.939365 (+ 568us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20260806 09:28:04.965709 16836 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:04.872755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59866 (local address 127.16.86.188:44201)
0806 09:28:04.872972 (+ 217us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:04.872980 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:04.887896 (+ 14916us) server_negotiation.cc:407] Connection header received
0806 09:28:04.888415 (+ 519us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:04.888426 (+ 11us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:04.905649 (+ 17223us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:04.905869 (+ 220us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:04.910132 (+ 4263us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.912139 (+ 2007us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.917464 (+ 5325us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:04.918506 (+ 1042us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:04.918665 (+ 159us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:04.951312 (+ 32647us) server_negotiation.cc:365] Received SASL_INITIATE NegotiatePB request
0806 09:28:04.951325 (+ 13us) server_negotiation.cc:881] Received SASL_INITIATE request from client
0806 09:28:04.951333 (+ 8us) server_negotiation.cc:892] Client requested to use mechanism: GSSAPI
0806 09:28:04.951427 (+ 94us) server_negotiation.cc:922] calling sasl_server_start()
0806 09:28:04.958891 (+ 7464us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.960539 (+ 1648us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.960552 (+ 13us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.960559 (+ 7us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.961164 (+ 605us) server_negotiation.cc:377] Sending SASL_CHALLENGE NegotiatePB response
0806 09:28:04.962166 (+ 1002us) server_negotiation.cc:365] Received SASL_RESPONSE NegotiatePB request
0806 09:28:04.962174 (+ 8us) server_negotiation.cc:956] Received SASL_RESPONSE request from client
0806 09:28:04.962177 (+ 3us) server_negotiation.cc:967] calling sasl_server_step()
0806 09:28:04.962744 (+ 567us) server_negotiation.cc:377] Sending SASL_SUCCESS NegotiatePB response
0806 09:28:04.963180 (+ 436us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:04.964541 (+ 1361us) server_negotiation.cc:299] Negotiation successful
0806 09:28:04.965115 (+ 574us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20260806 09:28:04.965746 16839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0683645ca7c41758405ff21832a4f25. Latest consensus state: current_term: 1 leader_uuid: "d0683645ca7c41758405ff21832a4f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } } }
I20260806 09:28:04.966179 16839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:28:04.972455 16866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0683645ca7c41758405ff21832a4f25. Latest consensus state: current_term: 1 leader_uuid: "d0683645ca7c41758405ff21832a4f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } } }
I20260806 09:28:04.972872 16866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:28:04.893268 16512 master_runner.cc:186] Error getting master registration for 127.16.86.188:44201: OK, instance_id { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" instance_seqno: 1786008484652626 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260806 09:28:04.977463 16512 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20260806 09:28:04.978574 16909 catalog_manager.cc:795] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260806 09:28:04.985563 16839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0683645ca7c41758405ff21832a4f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } } }
I20260806 09:28:04.986112 16839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:28:04.988183 16905 mvcc.cc:205] Tried to move back new op lower bound from 7315490754213126144 to 7315490753866813440. Current Snapshot: MvccSnapshot[applied={T|T < 7315490754213126144}]
I20260806 09:28:04.996402 16866 sys_catalog.cc:455] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d0683645ca7c41758405ff21832a4f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } } }
I20260806 09:28:04.996815 16866 sys_catalog.cc:458] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08 [sys.catalog]: This master's current role is: FOLLOWER
I20260806 09:28:04.997536 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0683645ca7c41758405ff21832a4f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } } }
I20260806 09:28:04.999243 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [sys.catalog]: This master's current role is: LEADER
I20260806 09:28:05.002056 16856 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d0683645ca7c41758405ff21832a4f25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d0683645ca7c41758405ff21832a4f25" member_type: VOTER last_known_addr { host: "127.16.86.190" port: 46357 } } peers { permanent_uuid: "0d95fd205b684662b5d51ad3dc927d83" member_type: VOTER last_known_addr { host: "127.16.86.189" port: 46825 } } peers { permanent_uuid: "582483e4cf4c4ce7a487b7c813931b08" member_type: VOTER last_known_addr { host: "127.16.86.188" port: 44201 } } }
I20260806 09:28:05.002645 16856 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25 [sys.catalog]: This master's current role is: LEADER
I20260806 09:28:05.004367 16878 catalog_manager.cc:1347] Generated new cluster ID: 2bf32b348ed44490a9ec1240f8727ecc
I20260806 09:28:05.004524 16878 catalog_manager.cc:1495] Initializing Kudu internal certificate authority...
I20260806 09:28:05.023926 16878 catalog_manager.cc:1370] Generated new certificate authority record
I20260806 09:28:05.025956 16878 catalog_manager.cc:1504] Loading token signing keys...
I20260806 09:28:05.040897 16878 catalog_manager.cc:5868] T 00000000000000000000000000000000 P d0683645ca7c41758405ff21832a4f25: Generated new TSK 0
I20260806 09:28:05.042224 16878 catalog_manager.cc:1514] Initializing in-progress tserver states...
W20260806 09:28:05.153923 16832 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.149791 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34088 (local address 127.16.86.190:46357)
0806 09:28:05.150159 (+ 368us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.150170 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.151164 (+ 994us) server_negotiation.cc:407] Connection header received
0806 09:28:05.151622 (+ 458us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.151630 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.151796 (+ 166us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.151899 (+ 103us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.153553 (+ 1654us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34088: BlockingRecv error: recv got EOF from 127.0.0.1:34088 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260806 09:28:05.154033 16835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.149875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38508 (local address 127.16.86.189:46825)
0806 09:28:05.150158 (+ 283us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.150164 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.151770 (+ 1606us) server_negotiation.cc:407] Connection header received
0806 09:28:05.152086 (+ 316us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.152093 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.152200 (+ 107us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.152285 (+ 85us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.153734 (+ 1449us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38508: BlockingRecv error: recv got EOF from 127.0.0.1:38508 (error 108)
Metrics: {"server-negotiator.queue_time_us":87}
W20260806 09:28:05.157033 16836 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.152270 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59878 (local address 127.16.86.188:44201)
0806 09:28:05.152520 (+ 250us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.152526 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.154805 (+ 2279us) server_negotiation.cc:407] Connection header received
0806 09:28:05.155127 (+ 322us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.155134 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.155240 (+ 106us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.155335 (+ 95us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.156706 (+ 1371us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59878: BlockingRecv error: recv got EOF from 127.0.0.1:59878 (error 108)
Metrics: {"server-negotiator.queue_time_us":121}
W20260806 09:28:05.164147 16835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.161288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38518 (local address 127.16.86.189:46825)
0806 09:28:05.161638 (+ 350us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.161646 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.162159 (+ 513us) server_negotiation.cc:407] Connection header received
0806 09:28:05.162410 (+ 251us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.162418 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.162592 (+ 174us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.162763 (+ 171us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.163805 (+ 1042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38518: BlockingRecv error: recv got EOF from 127.0.0.1:38518 (error 108)
Metrics: {"server-negotiator.queue_time_us":146}
W20260806 09:28:05.172806 16835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.170252 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38520 (local address 127.16.86.189:46825)
0806 09:28:05.170494 (+ 242us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.170502 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.170861 (+ 359us) server_negotiation.cc:407] Connection header received
0806 09:28:05.171091 (+ 230us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.171098 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.171242 (+ 144us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.171405 (+ 163us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.172350 (+ 945us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38520: BlockingRecv error: recv got EOF from 127.0.0.1:38520 (error 108)
Metrics: {"server-negotiator.queue_time_us":93}
W20260806 09:28:05.181785 16835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.178971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.16.86.189:46825)
0806 09:28:05.179324 (+ 353us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.179331 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.179842 (+ 511us) server_negotiation.cc:407] Connection header received
0806 09:28:05.180118 (+ 276us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.180123 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.180217 (+ 94us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.180332 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.181438 (+ 1106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38528: BlockingRecv error: recv got EOF from 127.0.0.1:38528 (error 108)
Metrics: {"server-negotiator.queue_time_us":131}
W20260806 09:28:05.194428 16835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.191782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.16.86.189:46825)
0806 09:28:05.192099 (+ 317us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.192106 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.192502 (+ 396us) server_negotiation.cc:407] Connection header received
0806 09:28:05.192838 (+ 336us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.192846 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.192957 (+ 111us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.193127 (+ 170us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.194160 (+ 1033us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38536: BlockingRecv error: recv got EOF from 127.0.0.1:38536 (error 108)
Metrics: {"server-negotiator.queue_time_us":147}
W20260806 09:28:05.212857 16835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.208380 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38540 (local address 127.16.86.189:46825)
0806 09:28:05.208719 (+ 339us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.208731 (+ 12us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.209473 (+ 742us) server_negotiation.cc:407] Connection header received
0806 09:28:05.210256 (+ 783us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.210262 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.210401 (+ 139us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.210516 (+ 115us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.212468 (+ 1952us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38540: BlockingRecv error: recv got EOF from 127.0.0.1:38540 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
W20260806 09:28:05.237330 16835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.234545 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.16.86.189:46825)
0806 09:28:05.234866 (+ 321us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.234874 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.235365 (+ 491us) server_negotiation.cc:407] Connection header received
0806 09:28:05.235777 (+ 412us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.235785 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.235910 (+ 125us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.236072 (+ 162us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.237047 (+ 975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38550: BlockingRecv error: recv got EOF from 127.0.0.1:38550 (error 108)
Metrics: {"server-negotiator.queue_time_us":132}
W20260806 09:28:05.277024 16835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.274458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38554 (local address 127.16.86.189:46825)
0806 09:28:05.274760 (+ 302us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.274767 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.275247 (+ 480us) server_negotiation.cc:407] Connection header received
0806 09:28:05.275465 (+ 218us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.275474 (+ 9us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.275606 (+ 132us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.275773 (+ 167us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.276745 (+ 972us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38554: BlockingRecv error: recv got EOF from 127.0.0.1:38554 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260806 09:28:05.349817 16835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.347213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38560 (local address 127.16.86.189:46825)
0806 09:28:05.347508 (+ 295us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.347516 (+ 8us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.347948 (+ 432us) server_negotiation.cc:407] Connection header received
0806 09:28:05.348165 (+ 217us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.348175 (+ 10us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.348290 (+ 115us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.348455 (+ 165us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.349512 (+ 1057us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38560: BlockingRecv error: recv got EOF from 127.0.0.1:38560 (error 108)
Metrics: {"server-negotiator.queue_time_us":102}
W20260806 09:28:05.485363 16835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.482910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38572 (local address 127.16.86.189:46825)
0806 09:28:05.483277 (+ 367us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.483284 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.483615 (+ 331us) server_negotiation.cc:407] Connection header received
0806 09:28:05.483919 (+ 304us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.483924 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.484019 (+ 95us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.484149 (+ 130us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.485099 (+ 950us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38572: BlockingRecv error: recv got EOF from 127.0.0.1:38572 (error 108)
Metrics: {"server-negotiator.queue_time_us":154}
W20260806 09:28:05.749189 16835 negotiation.cc:337] Failed RPC negotiation. Trace:
0806 09:28:05.747221 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.16.86.189:46825)
0806 09:28:05.747472 (+ 251us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:05.747478 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:05.747504 (+ 26us) server_negotiation.cc:407] Connection header received
0806 09:28:05.747609 (+ 105us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:05.747615 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:05.747716 (+ 101us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:05.747889 (+ 173us) server_negotiation.cc:226] Negotiated authn=SASL
0806 09:28:05.748903 (+ 1014us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38584: BlockingRecv error: recv got EOF from 127.0.0.1:38584 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20260806 09:28:05.807880 16857 catalog_manager.cc:1259] Loaded cluster ID: 2bf32b348ed44490a9ec1240f8727ecc
I20260806 09:28:05.808097 16857 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83: loading cluster ID for follower catalog manager: success
I20260806 09:28:05.811911 16857 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83: acquiring CA information for follower catalog manager: success
I20260806 09:28:05.814581 16857 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 0d95fd205b684662b5d51ad3dc927d83: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:28:05.984441 16909 catalog_manager.cc:1259] Loaded cluster ID: 2bf32b348ed44490a9ec1240f8727ecc
I20260806 09:28:05.984598 16909 catalog_manager.cc:1552] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08: loading cluster ID for follower catalog manager: success
I20260806 09:28:05.988821 16909 catalog_manager.cc:1574] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08: acquiring CA information for follower catalog manager: success
I20260806 09:28:05.991655 16909 catalog_manager.cc:1602] T 00000000000000000000000000000000 P 582483e4cf4c4ce7a487b7c813931b08: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260806 09:28:06.275887 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:06.266147 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38594 (local address 127.16.86.189:46825)
0806 09:28:06.266982 (+ 835us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:06.266989 (+ 7us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:06.267525 (+ 536us) server_negotiation.cc:407] Connection header received
0806 09:28:06.267936 (+ 411us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:06.267944 (+ 8us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:06.268225 (+ 281us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:06.268402 (+ 177us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:28:06.269081 (+ 679us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:06.269967 (+ 886us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:06.272629 (+ 2662us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:06.273183 (+ 554us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:06.273309 (+ 126us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:06.274092 (+ 783us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:28:06.274866 (+ 774us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:28:06.275034 (+ 168us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:06.275216 (+ 182us) server_negotiation.cc:299] Negotiation successful
0806 09:28:06.275336 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":410,"threads_started":1}
I20260806 09:28:06.293088 17024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:06.284782 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59884 (local address 127.16.86.188:44201)
0806 09:28:06.285930 (+ 1148us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:06.285936 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:06.285959 (+ 23us) server_negotiation.cc:407] Connection header received
0806 09:28:06.286088 (+ 129us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:06.286093 (+ 5us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:06.286331 (+ 238us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:06.286569 (+ 238us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:28:06.287237 (+ 668us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:06.288485 (+ 1248us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:06.290036 (+ 1551us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:06.290533 (+ 497us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:06.290686 (+ 153us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:06.291414 (+ 728us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:28:06.292215 (+ 801us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:28:06.292417 (+ 202us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:06.292489 (+ 72us) server_negotiation.cc:299] Negotiation successful
0806 09:28:06.292632 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":937,"thread_start_us":680,"threads_started":1}
I20260806 09:28:06.323709 17033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:06.301562 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34100 (local address 127.16.86.190:46357)
0806 09:28:06.312810 (+ 11248us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:06.312821 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:06.313995 (+ 1174us) server_negotiation.cc:407] Connection header received
0806 09:28:06.314364 (+ 369us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:06.314371 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:06.314623 (+ 252us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:06.314866 (+ 243us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:28:06.316215 (+ 1349us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:06.317494 (+ 1279us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:06.320012 (+ 2518us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:06.320661 (+ 649us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:06.320800 (+ 139us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:06.321840 (+ 1040us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:28:06.322350 (+ 510us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:28:06.322477 (+ 127us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:06.323222 (+ 745us) server_negotiation.cc:299] Negotiation successful
0806 09:28:06.323326 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11003,"thread_start_us":11280,"threads_started":1}
I20260806 09:28:06.324550 17014 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:06.301039 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.16.86.189:46825)
0806 09:28:06.301305 (+ 266us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:06.301316 (+ 11us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:06.314857 (+ 13541us) server_negotiation.cc:407] Connection header received
0806 09:28:06.315232 (+ 375us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:06.315239 (+ 7us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:06.315363 (+ 124us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:06.315440 (+ 77us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:28:06.316787 (+ 1347us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:06.318404 (+ 1617us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:06.321967 (+ 3563us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:06.322676 (+ 709us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:06.322775 (+ 99us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:06.323533 (+ 758us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:28:06.323691 (+ 158us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:28:06.323759 (+ 68us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:06.324075 (+ 316us) server_negotiation.cc:299] Negotiation successful
0806 09:28:06.324183 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20260806 09:28:06.329825 17024 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
0806 09:28:06.315388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59894 (local address 127.16.86.188:44201)
0806 09:28:06.315708 (+ 320us) server_negotiation.cc:206] Beginning negotiation
0806 09:28:06.315714 (+ 6us) server_negotiation.cc:399] Waiting for connection header
0806 09:28:06.322632 (+ 6918us) server_negotiation.cc:407] Connection header received
0806 09:28:06.323181 (+ 549us) server_negotiation.cc:365] Received NEGOTIATE NegotiatePB request
0806 09:28:06.323187 (+ 6us) server_negotiation.cc:461] Received NEGOTIATE request from client
0806 09:28:06.323316 (+ 129us) server_negotiation.cc:377] Sending NEGOTIATE NegotiatePB response
0806 09:28:06.323406 (+ 90us) server_negotiation.cc:226] Negotiated authn=TOKEN
0806 09:28:06.324275 (+ 869us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:06.325400 (+ 1125us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:06.327648 (+ 2248us) server_negotiation.cc:365] Received TLS_HANDSHAKE NegotiatePB request
0806 09:28:06.328306 (+ 658us) server_negotiation.cc:377] Sending TLS_HANDSHAKE NegotiatePB response
0806 09:28:06.328468 (+ 162us) server_negotiation.cc:657] Negotiated auth-only TLSv1.2 with cipher ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 Kx=ECDH Au=RSA Enc=AESGCM(128) Mac=AEAD
0806 09:28:06.328907 (+ 439us) server_negotiation.cc:365] Received TOKEN_EXCHANGE NegotiatePB request
0806 09:28:06.329050 (+ 143us) server_negotiation.cc:377] Sending TOKEN_EXCHANGE NegotiatePB response
0806 09:28:06.329160 (+ 110us) server_negotiation.cc:1035] Waiting for connection context
0806 09:28:06.329319 (+ 159us) server_negotiation.cc:299] Negotiation successful
0806 09:28:06.329413 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20260806 09:28:06.338043 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 16512
I20260806 09:28:06.339392 16597 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:28:06.474807 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 16512
I20260806 09:28:06.504408 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 16614
I20260806 09:28:06.505965 16703 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:28:06.637735 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 16614
I20260806 09:28:06.666009 16730 external_mini_cluster.cc:1636] Attempting to check leaks for /tmp/dist-test-taskKAHevx/build/asan/bin/kudu pid 16728
I20260806 09:28:06.667682 16818 generic_service.cc:196] Checking for leaks (request via RPC)
I20260806 09:28:06.793265 16730 external_mini_cluster.cc:1595] Killing /tmp/dist-test-taskKAHevx/build/asan/bin/kudu with pid 16728
2026-08-06T09:28:06Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5995 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (11435 ms total)
[----------] Global test environment tear-down
[==========] 56 tests from 4 test suites ran. (425216 ms total)
[ PASSED ] 55 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] SecurityITest.TestJwtMiniCluster
1 FAILED TEST